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Preface 


The  National  Science  Foundation’s  Math  and  Science  Partnership  program  supports  partner¬ 
ships  that  unite  the  efforts  of  K-12  school  districts  with  those  of  educators  in  science,  math¬ 
ematics,  engineering,  and  education  departments  at  colleges  and  universities.  In  2003,  the 
Allegheny  Intermediate  Unit  (AIU)  was  awarded  funding  under  this  program  to  create 
the  Math  Science  Partnership  of  Southwest  Pennsylvania  (MSP).  The  MSP  brings  together 
K-12  school  districts,  intermediate  units,  and  institutions  of  higher  education  in  Southwestern 
Pennsylvania.  Its  goals  are  to  increase  K-12  students’  knowledge  of  mathematics  and  science, 
increase  the  quality  of  the  K-16  educator  workforce,  and  create  sustainable  coordination  of 
partnerships  in  the  intermediate  units.  Initially  funded  for  five  years,  the  project  end  date  was 
subsequently  extended. 

AIU  subcontracted  with  RAND  Education,  a  unit  of  the  RAND  Corporation,  to  join  an 
evaluation  team  that  also  includes  the  Collaborative  for  Evaluation  and  Assessment  Capacity 
at  the  University  of  Pittsburgh  and  AIU’s  own  evaluation  division.  Collectively,  this  team  is 
investigating  the  effectiveness  of  the  partnership,  its  impact  on  practices  and  policies  at  part¬ 
ner  educational  institutions,  changes  in  math  and  science  instruction,  and  changes  in  student 
course  taking  and  achievement. 

The  five-year  term  of  RAND’s  subcontract  ended  in  2008.  This  monograph  describes 
findings  to  date  regarding  the  project’s  progress  toward  its  goals.  Notably,  it  does  not  constitute 
a  final  evaluation  of  the  MSP  because  the  project  continues  to  operate. 

This  monograph  should  be  of  interest  to  educators  and  policymakers  who  are  consider¬ 
ing  or  engaged  in  large-scale  efforts  to  improve  mathematics  and  science  education,  evaluators 
interested  in  methodologies  used  to  assess  the  value  of  large-scale  reforms,  and  institutions  of 
higher  education  that  may  be  interested  in  partnership  efforts  with  K-12  school  districts.  The 
evaluation  is  part  of  a  larger  body  of  RAND  Education  work  addressing  teachers  and  teaching, 
mathematics  and  science  achievement,  and  educational  reforms. 


Contents 


Preface . iii 

Figures .  ix 

Tables .  xi 

Summary .  XV 

Acknowledgments . xxi 

Abbreviations .  xxiii 

CHAPTER  ONE 

Introduction .  1 

Overview  of  rhe  Math  and  Science  Partnership  Program .  1 

Overview  of  the  Math  Science  Partnership  of  Southwest  Pennsylvania . 3 

Purpose  of  This  Monograph . 4 

Organization  of  This  Monograph . 4 

CHAPTER  TWO 

Math  Science  Partnership  of  Southwest  Pennsylvania . 5 

Intervention  Strategies . 5 

Professional  Development  for  Content  and  Leadership . 5 

Curriculum  Alignment  and  Pedagogical  and  Course  Refinement . 8 

Support  for  and  Dissemination  of  Research-Based  Materials . 9 

Motivation  for  the  MSP . 9 

Organizational  Structure  and  Management  of  the  Partnership . 10 

CHAPTER  THREE 

Evaluation  Design . 13 

Evaluation  Questions . 13 

Logic  Model . 13 

Resources  or  Inputs . 15 

Intervention  Strategies  and  MSP  Activities . 15 

Outputs . 15 

Short-Term  Outcomes . 15 

Midterm  Outcomes . 15 

Long-Term  Outcomes . 16 

Theory  of  Action . 16 

Evaluation  Data  and  Analyses . 17 

Limitations  of  This  Evaluation . 18 


v 


vi  Math  Science  Partnership  of  Southwest  Pennsylvania:  Measuring  Progress  Toward  Goals 


CHAPTER  FOUR 

Student  Achievement . 19 

Measures  of  MSP  Participation . 19 

Participation  Database . 19 

Development  of  Participation  Measures  for  Analysis .  20 

Limitations  of  the  Participation  Measures . 21 

Measures  of  Student  Achievement . 21 

Mathematics  Assessments . 21 

Science  Assessments .  22 

Student  Test  Score  Database .  22 

Analytic  Approaches  to  Relating  Student  Achievement  to  MSP  Participation .  23 

Student-Level  Achievement  Models .  23 

District-Level  Achievement  Models .  24 

Statewide  Comparison  Group  Achievement  Models .  24 

Descriptive  Summary  of  Participation  by  K-12  Teachers  and  Leaders .  26 

Trends  in  Mathematics  and  Science  Achievement . 29 

Relationship  of  MSP  Participation  to  Student  Achievement . 31 

Relationship  of  Participation  to  Math  Achievement . 31 

Relationship  of  Participation  to  Science  Achievement . 32 

Summary  of  Findings  Related  to  Student  Achievement . 33 

CHAPTER  FIVE 

Quality  of  the  Educator  Workforce . 35 

Survey  Development  and  Analysis . 35 

Principal  Survey  Scales . 37 

Teacher  Survey  Scales . 37 

Approaches  to  Survey  Analysis . 37 

Findings  Reported  by  K-12  Educators .  40 

Overall  Impact  of  the  MSP .  40 

Principal  Leadership .  40 

District  and  School  Policies  and  Practices .  42 

Awareness  and  Knowledge  of  Research-Based  Instructional  Practices  and  Materials .  42 

Creation  of  Professional  Learning  Communities .  44 

Instructional  Practices  of  K-12  Teachers .  44 

Analysis  of  the  Relationship  of  MSP  Participation  to  Survey  Results . 45 

Contribution  of  IHEs  to  the  Quality  of  the  Educator  Workforce .  46 

Classroom  Instructional  Practices  at  IHEs .  46 

Practices  Related  to  Student-Teacher  Placements .  48 

Summary  of  Findings  Related  to  the  Quality  of  the  Educator  Workforce .  50 

CHAPTER  SIX 

Sustainable  Partnerships . 53 

Partnership  Development . 53 

Conceptualizations  of  Partnership .  54 

Partnership  Development  Between  K— 12  and  IHEs . 55 

Partnership  Development  Across  IHEs . 57 


Contents  vii 


Partnership  Development  Across  Departments  Within  IHEs . 57 

Changes  in  Institutional  Policies  to  Support  Sustainability . 59 

IHE  Academic  Reward  Systems . 59 

Changes  to  IE1E  Recognition  and  Reward  Policies . 61 

Informal  Rewards  for  MSP  Participation . 62 

Implementing  Challenging  Courses:  IEIE  Course  Revisions . 63 

Summary  of  Findings  Related  to  Sustainable  Partnership .  64 

CHAPTER  SEVEN 

Conclusion . 67 

APPENDIXES 

A.  Achievement  Analysis  Methods  and  Results . 69 

B.  Teacher  and  Principal  Survey  Development  and  Methods .  Ill 

C.  Qualitative  Analysis  of  IHE  Data .  117 

References .  121 


Figures 


3.1.  MSP  Logic  Model . 14 

4.1.  Trends  in  the  Number  of  Participants  in  MSP  Activities .  26 

4.2.  Trends  in  Average  Time  Spent  by  Participants  on  MSP  Activities .  27 

4.3.  Participation  in  Math  and  Science  MSP  Activities  by  District  Leaders .  28 

4.4.  Adjusted  Participation  in  Math  and  Science  MSP  Activities  by  Teachers .  28 

4.5.  Proficiency  Levels  in  MSP  and  Non-MSP  Districts .  30 

6.1.  Partnership  Development  Between  IHE  Faculty  and  K-12  Educators  Who 

Participated  in  the  Teacher  Fellow  Program .  56 

6.2.  Partnership  Development  Across  IFlEs . 57 

6.3.  Partnership  Development  Between  Departments  at  One  IHE . 58 

A.l.  Representativeness  of  Districts  Included  in  Student-Level  Models  in  Terms  of 

Leader  Participation . 70 

A. 2.  Representativeness  of  Districts  Included  in  Student-Level  Models  in  Terms  of 

Mathematics  Teacher  Participation . 71 

A. 3.  Representativeness  of  Districts  Included  in  Student-Level  Models  in  Terms  of 

Science  Teacher  Participation . 71 

A.4.  Representativeness  of  Schools  Included  in  Student-Level  Models  in  Terms  of 

Mathematics  Teacher  Participation . 72 

A. 5.  Representativeness  of  Schools  Included  in  Student-Level  Models  in  Terms  of 

Science  Teacher  Participation . 72 


IX 


Tables 


5.1.  Primary  Activities  of  the  MSP . xvi 

2.1.  Primary  Activities  of  the  MSP . 6 

4.1.  Math  and  Science  Achievement  Scores,  by  Year  and  Cohort .  23 

4.2.  PSSA  Mathematics  Proficiency  Levels  in  MSP  and  Non-MSP  Districts . 29 

4.3.  MSP  District  Science  Assessment  Scores . 31 

4.4.  Summary  of  Mathematics  Achievement  Models . 32 

4.5.  Summary  of  Science  Achievement  Models . 32 

5.1.  Principal  Survey  Scales .  38 

5.2.  Teacher  Survey  Scales . 39 

5.3.  Teacher  Responses  Regarding  the  Impact  of  MSP  Activities  in  Which  They 

Participated . 41 

5.4.  Principal  Responses  Regarding  Leadership  Skills .  42 

5.5.  Teacher  Responses  Regarding  Principal  Leadership .  42 

5.6.  Principal  Responses  Regarding  School  and  District  Policies  and  Practices .  43 

5.7.  Teacher  Responses  Regarding  Classroom  Practices .  44 

5.8.  Teacher  Responses  Regarding  Professional  Development .  44 

5.9.  Teacher  Responses  Regarding  Student  Learning  Activities  . 45 

6.1.  Patterns  of  Partnership  Development  Between  IHE  Departments . 58 

6.2.  Factors  Associated  with  Partnership  Development  Between  Institutions . 59 

A.l.  Descriptive  Statistics  of  Variables  Included  in  the  Student-Level  Mixed-Effect 

Model  Analysis  for  Mathematics,  Cohort  G  (Grade  11  in  2006-07) . 75 

A. 2.  Student-Level  Mixed-Effect  Model  Analysis  Results  for  Mathematics,  Cohort  G 

(Grade  11  in  2006-07) . 75 

A. 3.  Descriptive  Statistics  of  Variables  Included  in  the  Student-Level  Mixed-Effect 

Model  Analysis  for  Mathematics,  Cohort  K  (Grade  7  in  2006-07) . 76 

A.4.  Student-Level  Mixed-Effect  Model  Analysis  Results  for  Mathematics,  Cohort  K 

(Grade  7  in  2006-07) . 76 

A. 5.  Descriptive  Statistics  of  Variables  Included  in  the  Student-Level  Mixed-Effect 

Model  Analysis  for  Mathematics,  Cohort  L  (Grade  6  in  2006-07) .  77 

A.6.  Student-Level  Mixed-Effect  Model  Analysis  Results  for  Mathematics,  Cohort  L 

(Grade  6  in  2006-07) .  77 

A. 7.  Descriptive  Statistics  of  Variables  Included  in  the  Student-Level  Mixed-Effect 

Model  Analysis  for  Mathematics,  Cohort  N  (Grade  4  in  2006-07) . 78 

A.  8.  Student-Level  Mixed-Effect  Model  Analysis  Results  for  Mathematics,  Cohort  N 

(Grade  4  in  2006-07) . 78 

A.9.  Descriptive  Statistics  of  Variables  Included  in  District-Level  Regression  Analysis 

for  Mathematics,  Cohort  G  (Grade  11  in  2006-07) . 79 


XI 


xii  Math  Science  Partnership  of  Southwest  Pennsylvania:  Measuring  Progress  Toward  Goals 


A.  10.  District-Level  Regression  Results  for  Mathematics,  Cohort  G 

(Grade  11  in  2006-07) . 79 

A.ll.  Descriptive  Statistics  of  Variables  Included  in  District-Level  Regression  Analysis 

for  Mathematics,  Cohort  J  (Grade  8  in  2006-07) .  80 

A.12.  District-Level  Regression  Results  for  Mathematics,  Cohort  J  (Grade  8  in  2006-07). ...  80 
A. 13.  Descriptive  Statistics  of  Variables  Included  in  District-Level  Regression  Analysis 

for  Mathematics,  Cohort  K  (Grade  7  in  2006-07) . 81 

A.  14.  District-Level  Regression  Results  for  Mathematics,  Cohort  K  (Grade  7  in  2006-07). ...  81 
A. 15.  Descriptive  Statistics  of  Variables  Included  in  District-Level  Regression  Analysis 

for  Mathematics,  Cohort  L  (Grade  6  in  2006-07) . 82 

A. 16.  District-Level  Regression  Results  for  Mathematics,  Cohort  L  (Grade  6  in  2006-07) ....  82 
A.  17.  Descriptive  Statistics  of  Variables  Included  in  District-Level  Regression  Analysis 

for  Mathematics,  Cohort  M  (Grade  5  in  2006-07) . 83 

A. 18.  District-Level  Regression  Results  for  Mathematics,  Cohort  M  (Grade  5  in  2006-07) ...  83 
A.  19.  Descriptive  Statistics  of  Variables  Included  in  District-Level  Regression  Analysis 

for  Mathematics,  Cohort  N  (Grade  4  in  2006-07) .  84 

A. 20.  District-Level  Regression  Results  for  Mathematics,  Cohort  N 

(Grade  4  in  2006-07) .  84 

A.21.  Descriptive  Statistics  of  Variables  Included  in  District-Level  Regression  Analysis 

for  Science,  Cohort  F  (Grade  10  in  2004-05)  . 85 

A. 22.  District-Level  Regression  Results  for  Science,  Cohort  F  (Grade  10  in  2004-05)  . 85 

A. 23.  Descriptive  Statistics  of  Variables  Included  in  District-Level  Regression  Analysis 

for  Science,  Cohort  G  (Grade  10  in  2005-06)  .  86 

A. 24.  District-Level  Regression  Results  for  Science,  Cohort  G  (Grade  10  in  2005-06) .  86 

A. 25.  Descriptive  Statistics  of  Variables  Included  in  District-Level  Regression  Analysis 

for  Science,  Cohort  Ff  (Grade  10  in  2006-07) .  87 

A. 26.  District-Level  Regression  Results  for  Science,  Cohort  Ff  (Grade  10  in  2006-07)  .  87 

A. 27.  Descriptive  Statistics  of  Variables  Included  in  District-Level  Regression  Analysis 

for  Science,  Cohort  I  (Grade  7  in  2004-05) .  88 

A.28.  District-Level  Regression  Results  for  Science,  Cohort  I  (Grade  7  in  2004-05)  .  88 

A. 29.  Descriptive  Statistics  of  Variables  Included  in  District-Level  Regression  Analysis 

for  Science,  Cohort  J  (Grade  7  in  2005-06) . 89 

A.30.  District-Level  Regression  Results  for  Science,  Cohort  J  (Grade  7  in  2005-06) . 89 

A. 31.  Descriptive  Statistics  of  Variables  Included  in  District-Level  Regression  Analysis 

for  Science,  Cohort  K  (Grade  7  in  2006-07) .  90 

A. 32.  District-Level  Regression  Results  for  Science,  Cohort  K  (Grade  7  in  2006-07) .  90 

A. 33.  Descriptive  Statistics  of  Variables  Included  in  District-Level  Regression  Analysis 

for  Science,  Cohort  M  (Grade  4  in  2005-06) . 91 

A. 34.  District-Level  Regression  Results  for  Science,  Cohort  M  (Grade  4  in  2005-06) . 91 

A. 35.  Descriptive  Statistics  of  Variables  Included  in  District-Level  Regression  Analysis 

for  Science,  Cohort  N  (Grade  4  in  2006-07) .  92 

A. 36.  District-Level  Regression  Results  for  Science,  Cohort  N  (Grade  4  in  2006-07) .  92 

A. 37.  Balance  in  Covariates,  Before  and  After  Matching,  Comparison  Group  1 . 93 

A.38.  Balance  in  Covariates,  Before  and  After  Matching,  Comparison  Group  2 .  98 

A. 39.  Balance  in  Covariates,  Before  and  After  Matching,  Comparison  Group  3 .  103 

A.40.  State-Level  Regression  Results  for  Mathematics,  Cohort  G  (Grade  11  in  2006-07)  ...  108 

A.41.  State-Level  Regression  Results  for  Mathematics,  Cohort  J  (Grade  8  in  2006-07) .  109 


Tables  xiii 


B.l.  Teacher  Survey  Sample  and  Response  Rates .  112 

B. 2.  Principal  Survey  Respondent  Demographics .  113 

C. l.  IHE  Faculty  Member  Characteristics .  119 


Summary 


The  National  Science  Foundation’s  (NSF’s)  Math  and  Science  Partnership  program  began  in 
2002  with  the  goals  of  providing  a  challenging  curriculum  for  every  student,  increasing  the 
quality  and  diversity  of  mathematics  and  science  teachers,  engaging  a  network  of  researchers 
and  practitioners  in  partnership,  and  disseminating  research-based  materials.  In  2003,  the 
Allegheny  Intermediate  Unit  (AIU)  received  a  grant  under  this  program  to  establish  the  Math 
Science  Partnership  of  Southwest  Pennsylvania  (MSP). 

The  MSP  brings  together  53  K-12  school  districts  in  Southwestern  Pennsylvania;  four 
regional,  small-  to  medium-sized,  teaching-oriented,  private  institutions  of  higher  education 
(IHEs);  and  four  intermediate  units  (IUs).  Its  goals  are  to  increase  K-12  students’  knowledge 
of  mathematics  and  science,  improve  the  quality  of  the  K-16  educator  workforce,  and  create 
sustainable  coordination  of  partnerships  in  the  IUs.  Initially  funded  for  five  years,  the  project 
end  date  was  subsequently  extended  by  two  years. 

The  purpose  of  this  monograph  is  to  summarize  key  findings  regarding  the  project’s 
progress  toward  meeting  its  goals.  Analyses  draw  on  data  collected  through  the  MSP’s  first 
five  years  of  implementation,  information  from  surveys  of  teachers  and  principals,  qualita¬ 
tive  data  from  participating  IFIEs,  educator  participation  records,  and  assessments  of  student 
achievement. 


MSP  Intervention  Strategies 

The  MSP  used  three  crosscutting  intervention  strategies  to  accomplish  its  goals.  The  first  is 
professional  development  for  content  and  leadership  through  academies  and  seminars  for  K-12 
educators  and  IHE  faculty.  The  overriding  purpose  of  these  activities  is  to  equip  teachers  with 
the  content,  pedagogy,  and  leadership  skills  necessary  to  become  effective  leaders  in  their  insti¬ 
tutions.  The  second  strategy  is  curriculum  alignment  and  pedagogical  and  course  refinement, 
enacted  at  the  K-12  level  through  the  use  of  curriculum  frameworks  and  research-based  cur¬ 
riculum  materials  and  at  the  IHE  level  through  the  contributions  of  K-12  teachers  who  work 
with  faculty  on  IHE  campuses  to  refine  IHE  courses.  The  third  strategy  is  support  for  and  dis¬ 
semination  of  research-based  resources  and  tools,  primarily  through  conferences  and  networks 
connecting  educators  using  research-based  curricula.  Importantly,  these  intervention  strategies 
are  not  distinct  and  separable,  but  rather  are  intertwined  in  a  design  uniting  K-12  and  IHE 
educators  in  working  to  achieve  the  goals  of  the  MSP.  Table  S.l  describes  the  primary  activities 
used  by  the  MSP  to  enact  these  strategies. 


XV 


xvi  Math  Science  Partnership  of  Southwest  Pennsylvania:  Measuring  Progress  Toward  Goals 


Table  S.1 

Primary  Activities  of  the  MSP 


Intervention 

Strategy  MSP  Activity  Description 


Professional 
development 
for  content  and 
leadership 


Leadership  action  Leadership  action  teams  represent  each  school  district  and  IHE.  The 
teams  and  Leadership  teams  meet  at  the  Leadership  Action  Academies  to  assess  the  strengths 
Action  Academies  and  weaknesses  of  their  institutions  and  to  develop  action  plans  for 
improvement. 


Teacher  Leadership  Teacher  Leadership  Academies  provide  professional  development  on 
Academies  and  leadership  and  content  for  selected  teachers,  who  then  lead  on-site 
on-site  academies  academies  to  disseminate  the  MSP  professional  development  to  teachers 
in  their  own  districts. 


Principal  seminars  Seminars  for  principals  aim  to  build  a  deeper  understanding  of  effective 
mathematics  and  science  instruction  and  to  develop  effective  observing 
and  conferencing  techniques. 


Content  short 
courses 


Vouchers  and  stipends  support  teachers  who  attend  professional 
development  courses  in  math  or  science  content  areas  sponsored  by  IHE 
partners  and  others. 


Curriculum 
alignment  and 
pedagogical 
and  course 
refinement 


Teacher  fellows  The  teacher  fellow  program  provides  support  for  teachers  to  spend  one 
or  two  terms  at  a  partner  IHE  where  they  work  with  IHE  faculty  to  refine 
two  IHE  courses,  take  a  college  course,  and  participate  in  MSP  activities. 

Curriculum  The  MSP  developed  a  curriculum  framework  for  science  and  refined  the 

frameworks  one  for  math,  with  the  six  to  eight  "big  ideas"  to  be  taught  in  these 

disciplines  at  each  K-12  grade  level. 


Support  for  and 
dissemination  of 
research-based 
materials 


Network 

Connections 

conference 


This  daylong  conference  is  held  twice  per  year  for  leadership  action 
teams  and  other  math  and  science  teachers  and  IHE  faculty  to  explore 
research-based  resources  and  tools. 


Educator  networks  Networking  activities  assist  districts  in  implementing  challenging 

courses  and  curricula.  Groups  of  teachers  and  coaches  using  a  common 
curriculum  meet  to  share  best  practices. 


NOTE:  MSP  activities  are  shown  in  the  table  as  addressing  one  specific  intervention  strategy  on  which  they  are 
likely  to  have  the  greatest  overall  impact.  However,  most  actually  address  additional  intervention  strategies. 
For  example,  the  teacher  fellow  activity  addresses  not  only  curriculum  alignment  and  pedagogical  and  course 
refinement,  but  also  professional  development  for  content  and  leadership. 


Evaluation  Design 

The  evaluation  questions  addressed  in  this  monograph  are  based  on  the  MSP’s  goals: 

•  What  progress  has  been  made  during  the  implementation  of  the  MSP  toward  increas¬ 
ing  K-12  students’  math  and  science  achievement ,  and  to  what  extent  can  the  progress  be 
attributed  to  the  MSP? 

•  What  progress  has  been  made  during  the  implementation  of  the  MSP  toward  increasing 
the  quality  of  the  K—16  educator  workforce,  and  to  what  extent  can  the  progress  be  attrib¬ 
uted  to  the  MSP? 

•  What  progress  has  been  made  during  the  implementation  of  the  MSP  toward  creating 
sustainable  partnerships,  and  to  what  extent  can  the  progress  be  attributed  to  the  MSP? 

The  findings  reported  here  draw  on  evaluation  data  related  to  the  project’s  short-,  mid-, 
and  long-term  outcomes.  It  is  important  to  note  that  the  project  continues  to  collect  evaluation 
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data.  Partners  on  the  assessment  and  evaluation  team  are  collecting  data  from  case  studies  of 
K-12  school  districts,  observations  of  MSP  events,  interviews  with  key  project  personnel,  and 
assessments  of  teacher  learning  during  professional  development,  as  well  as  continuing  to  col¬ 
lect  and  analyze  student  achievement  data. 

This  monograph  describes  analyses  and  findings  related  to  the  three  evaluation  questions. 
These  analyses  rely  chiefly  on  mathematics  and  science  achievement  data  for  K-12  students, 
survey  data  from  K-12  educators,  and  IHE  qualitative  data.  In  addition  to  these  sources,  analy¬ 
ses  make  use  of  data  from  a  database  of  educator  participation  in  MSP  activities;  a  statewide 
database  on  school  district  demographics,  finances,  and  achievement;  and  the  project’s  entries 
in  an  NSF-sponsored  database  that  collects  information  on  all  math  and  science  partnership 
projects  annually. 

Analytic  approaches  to  addressing  the  evaluation  questions  can  be  summarized  as  fol¬ 
lows.  Changes  in  K-12  student  mathematics  and  science  test  scores  are  analyzed  using  three 
statistical  approaches  to  examine  the  relationship  of  those  changes  to  educator  participation  in 
the  project.  Changes  in  midterm  outcomes  associated  with  the  quality  of  the  K-16  educator 
workforce  are  examined  through  both  qualitative  analyses  of  IHE  data  and  statistical  analyses 
of  survey  data  to  examine  the  relationship  of  changes  to  educator  participation  in  the  project. 
Finally,  the  sustainability  of  partnerships  is  examined  using  indicators  drawn  from  qualitative 
analysis  of  IHE  data,  such  as  the  development  of  partnership,  the  implementation  of  challeng¬ 
ing  courses  through  mechanisms  established  by  the  MSP,  and  changes  in  institutional  policies 
and  practices. 


Descriptive  Summary  of  K-12  Participation 

Over  the  course  of  the  first  four  years  of  the  project,  58  percent  of  mathematics  and  science 
educators  in  the  MSP  districts  participated  in  project  activities.  Overall,  3,568  educators  par¬ 
ticipated  in  math-related  activities,  and  1,321  participated  in  science-related  activities,  though 
science  participation  was  increasing  in  the  later  years.  This  difference  between  math  and  sci¬ 
ence  may  be  due  to  the  ongoing  pressures  to  improve  mathematics  achievement,  along  with 
the  phase-in  of  lower-stakes  state  science  assessments  during  the  project.  On  average,  the  total 
number  of  hours  that  each  participant  spent  engaging  in  MSP  activities  over  the  four  years  was 
similar  for  the  two  subject  areas.  The  mean  total  individual  participation  was  25.1  hours  for 
math  and  2 7.6  hours  for  science  (the  medians  were  15.0  hours  for  math  and  10.3  hours  for  sci¬ 
ence).  Combining  math  and  science,  the  sum  of  individual-level  participation  over  four  years 
ranged  from  0.5  to  306  hours  but  was  in  the  range  of  five  to  50  hours  for  most  participants. 


Overall  Findings  on  MSP  Progress  Toward  Its  Goals 

Achievement  analyses  found  that  MSP  school  districts  experienced  trends  of  increase  in  stu¬ 
dent  mathematics  and  science  scores  during  the  project.  For  mathematics,  similar  trends  were 
observed  throughout  Pennsylvania;  for  science,  there  was  no  external  reference  for  comparison. 
Further  analyses  examined  the  relationship  between  educators’  MSP  participation  and  stu¬ 
dents’  math  and  science  achievement.  MSP  participation  measures  were  developed  to  account 
for  differences  in  the  potential  impact  of  educators  who  play  a  leadership  role,  as  opposed  to 
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educators  who  teach  but  do  not  play  a  leadership  role.  These  measures  were  then  used  in  three 
distinct  analytic  strategies  for  statistically  modeling  the  relationship  between  educator  par¬ 
ticipation  and  student  achievement.  Results  of  these  analyses  showed  only  a  few  significant 
relationships  between  MSP  participation  by  educators  and  student  math  achievement.  These 
significant  relationships  appeared  among  many  nonsignificant  findings  and  were  inconsistent 
across  cohorts  and  analyses — as  a  result,  they  do  not  enable  us  to  draw  any  overall  conclusion 
about  the  effects  of  the  MSP  on  changes  in  math  or  science  achievement. 

Analyses  of  K-12  survey  and  IHE  qualitative  data  examined  evidence  of  change  in  mid¬ 
term  outcomes  related  to  the  quality  of  the  K-16  educator  workforce.  Generally,  educators  at 
both  the  K-12  and  IHE  levels  reported  changes  in  leadership  and  instruction  that  are  con¬ 
sistent  with  the  MSP  theory  of  action.  K-12  teachers  reported  that  MSP  activities  increased 
their  awareness  and  understanding  of  math  and  science  concepts  and  how  students  think 
about  math  and  science  and  helped  them  change  their  teaching  practices.  They  also  reported 
that  the  professional  development  in  which  they  participated  was  more  often  relevant  to  their 
needs  and  was  focused  on  instructional  approaches  and  individual  student  learning.  Principals 
reported  that  the  principal  seminars  influenced  their  views  and  behaviors  as  principals.  IHE 
faculty  reported  a  greater  emphasis  on  student- centered  instruction  and  more  awareness  of  dif¬ 
ferent  pedagogical  techniques.  These  findings  suggest  that  the  activities  of  the  MSP  may  be 
having  the  intended  effect  on  midterm  outcomes  associated  with  increasing  the  quality  of  the 
educator  workforce.  However,  further  statistical  analyses  of  the  relationship  between  participa¬ 
tion  in  MSP  activities  and  survey  scales  tracking  key  midterm  outcomes  did  not  provide  evi¬ 
dence  that  the  MSP  is  responsible  for  the  changes  reported  by  educators.  Data  from  the  K-12 
case  studies  may  be  helpful  in  interpreting  these  findings. 

IHE  analyses  examined  the  development  of  sustainable  partnerships  in  the  MSP  and 
found  positive  indications  of  partnership  development  between  IHE  and  K-12  educators, 
between  IHEs,  and  between  departments  within  IHEs.  Moreover,  analyses  found  modest 
progress  toward  a  broader  definition  of  scholarship  in  IHE  faculty  reward  systems,  an  impor¬ 
tant  factor  in  partnership  development.  Participation  in  MSP  activities,  along  with  revisions 
to  IHE  courses  through  the  teacher  fellow  program,  helped  faculty  embrace  a  wider  variety  of 
approaches  to  presenting  the  material  to  their  students.  As  a  result  of  these  changes,  faculty 
members  were  optimistic  that  students  would  become  more  engaged  in  coursework  and  take 
responsibility  for  their  own  learning. 


Conclusion 

In  sum,  the  evaluation  found  numerous  indications  that  changes  are  occurring  that  are  consis¬ 
tent  with  the  MSP  theory  of  action.  Thus,  MSP  partners  appear  to  be  making  progress  toward 
the  three  MSP  goals,  though  attempts  to  statistically  link  this  progress  to  MSP  participation 
were  not  successful.  There  are  several  limitations  to  note.  The  evaluation  was  designed  to  be 
selective  in  its  data  collection  and  analyses,  primarily  assessing  the  project’s  achievement  of 
its  goals  and  the  major  pathways  toward  achieving  those  goals.  In  addition,  the  evaluation 
relies  on  self-report  data  from  a  sample  of  participants  and  is  subject  to  common  potential 
biases  associated  with  such  data.  Finally,  and  importantly,  if  the  MSP  intervention  strategies 
require  more  than  four  years  to  achieve  project  goals,  analyses  reported  here  may  not  detect  the 
impact.  This  monograph  concludes  RAND  Education’s  MSP  evaluation  activities;  however, 
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it  is  important  to  note  that  it  does  not  constitute  a  final  evaluation  of  the  MSP.  The  project, 
including  data  collection  and  evaluation  activities,  is  expected  to  continue  through  a  forecasted 
end  date  in  2010. 
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CHAPTER  ONE 


Introduction 


In  2003,  the  Allegheny  Intermediate  Unit  (AIU)  received  a  grant  from  the  National  Science 
Foundation  (NSF)  to  establish  the  Math  Science  Partnership  of  Southwest  Pennsylvania  (MSP). 
The  project’s  evaluation  team  comprises  the  Collaborative  for  Evaluation  and  Assessment 
Capacity  at  the  University  of  Pittsburgh,  AIU’s  own  evaluation  division,  and  RAND  Educa¬ 
tion,  a  unit  of  the  RAND  Corporation.  Collectively,  this  team  is  investigating  the  impact  of 
the  partnership  on  practices  and  policies  at  partner  educational  institutions,  changes  in  math 
and  science  instruction,  and  changes  in  student  course  taking  and  achievement. 

This  monograph  examines  the  MSP’s  progress,  through  five  years  of  implementation, 
toward  its  three  goals  of  increasing  K-12  students’  knowledge  of  mathematics  and  science, 
increasing  the  quality  of  the  K-16  educator  workforce,  and  creating  sustainable  coordina¬ 
tion  of  partnerships  in  the  intermediate  units  (IUs).1  This  chapter  provides  an  overview  of  the 
national  Math  and  Science  Partnership  initiative  funded  by  the  NSF,  followed  by  an  overview 
of  the  design  and  progress  of  the  MSP. 


Overview  of  the  Math  and  Science  Partnership  Program 

The  NSF’s  Math  and  Science  Partnership  program  began  as  an  initiative  of  the  No  Child  Left 
Behind  Act  of  2001  (NCLB),  largely  in  response  to  growing  concerns  about  the  ability  of  the 
United  States  to  remain  competitive  in  a  global  economy  with  continued  poor  performance 
among  students  in  math  and  science.  NCLB’s  education  reform  agenda  included  recommen¬ 
dations  that  the  following  three  issues  be  addressed:  too  many  teachers  teaching  out  of  their 
fields,  too  few  students  taking  advanced  coursework,  and  too  few  schools  offering  challenging 
curricula  and  textbooks  (NSF,  2002).  One  year  later,  Congress  established  the  Math  and  Sci¬ 
ence  Partnership  program  under  the  NSF  Authorization  Act  of  2002  to  focus  on  these  issues 
(see  Pub.  L.  107-368). 

The  Math  and  Science  Partnership  is  an  ambitious  program.  In  the  initial  program  solici¬ 
tation,  the  following  goals  were  identified  in  an  effort  to  improve  student  achievement: 

•  “[P]rovide  a  challenging  curriculum  for  every  student.” 

•  “[I]ncrease  and  sustain  the  number,  quality,  and  diversity”  of  K-12  mathematics  and  sci¬ 
ence  teachers  “through  further  development  of  a  professional  education  continuum.” 


1  Pennsylvania’s  IUs  are  publicly  funded  educational  service  agencies  that  serve  as  regional  intermediaries  between  locally 
controlled  school  districts  and  the  Pennsylvania  Department  of  Education. 
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•  “[CJontribute  to  the  national  capacity  to  engage  in  large-scale  reform  through  participa¬ 
tion  in  a  network  of  researchers  and  practitioners,  organized  through  the  [Math  and  Sci¬ 
ence  Partnership]  program.” 

•  “[E]ngage  the  learning  community  in  the  knowledge  base  being  developed  in  current  and 
future  NSF  Centers  for  Learning  and  Teaching  and  Science  of  Learning  Centers”  (NSF, 
2002). 

Although  the  language  in  subsequent  solicitations  evolved,  the  intent  to  implement  sweeping 
and  sustainable  change  has  remained. 

Prior  NSF  programs  targeting  math  and  science  educational  reform  have  had  similar 
goals.  For  example,  the  Collaborative  for  Excellence  in  Teacher  Preparation  program  sought 
to  lift  the  quality  of  education  received  by  preservice  teachers  by  involving  math,  science, 
and  engineering  departments  (Ruskus,  Matson,  and  Perakis,  2001),  and  the  Local  Systemic 
Change  program  focused  on  professional  development  for  in-service  K-12  teachers  of  math, 
science,  and  technology  (I.  Weiss,  Banilower,  et  ah,  2002).  However,  the  Math  and  Science 
Partnership  program  is  notable  in  requiring  equal  participation  of  K-12  and  institutions  of 
higher  education  (IHEs)  in  the  educational  reform.  Each  program  partnership  must  include 
one  or  more  school  districts  and  one  or  more  higher  education  entities  as  core  partners,  with 
additional  partners  encouraged  but  not  required.  Moreover,  the  Math  and  Science  Partner¬ 
ship  program  expects  full  participation  from  mathematics,  science,  and  engineering  faculty 
members  in  project  activities.  It  is  also  clear  that  the  NSF  expects  substantial  institutional 
change  to  occur  in  both  K-12  and  higher  education  and  that  it  intends  to  study  partnership 
models  to  learn  how  partners’  commitments  result  in  institutional  changes  that  lead  to  the 
scalability  and  sustainability  of  the  reform  efforts. 

Although  there  is  considerable  flexibility  in  the  designs  of  individual  projects,  they  are 
expected  to  incorporate  the  following  key  features: 

•  Partnerships:  Projects  should  be  designed  and  implemented  through  partnerships  that 
unite  administrators,  teachers,  and  guidance  counselors  in  K-12  partner  organizations,  as 
well  as  education  faculty  and  disciplinary  faculty  in  mathematics,  science,  and  engineer¬ 
ing  in  partner  IHEs. 

•  Teacher  quality,  quantity,  and  diversity:  Projects  should  enhance  and  sustain  the  number, 
quality,  and  racial/ethnic  diversity  of  K-12  mathematics  and  science  teachers. 

•  Challenging  courses  and  curricula :  Projects  should  ensure  that  K-12  students  are  prepared 
for,  have  access  to,  and  are  encouraged  to  participate  and  succeed  in  challenging  math¬ 
ematics  or  science  courses  and  curricula. 

•  Evidence-based  design  and  outcomes:  The  project  design  should  be  informed  by  the  cur¬ 
rent  literature  on  learning  and  teaching,  and  project  outcomes  should  promise  to  make 
evidence-based  contributions  to  the  learning  and  teaching  knowledge  base. 

•  Institutional  change  and  sustainability:  To  ensure  sustainability,  core  partner  organizations 
should  redirect  resources  and  design  and  should  implement  new  policies  and  practices 
that  result  in  well-documented,  inclusive,  and  coordinated  institutional  change  at  both 
the  K— 12  and  IHE  levels. 

To  date,  the  program  has  made  approximately  140  awards,  totaling  nearly  $700  million 
(NSF,  2008).  Funded  projects  have  involved  hundreds  of  IHEs  and  K-12  school  districts,  along 
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with  a  host  of  other  stakeholders.  Additional  projects  have  been  awarded  through  a  parallel 
Math  and  Science  Partnership  program  administered  by  the  U.S.  Department  of  Education, 
also  authorized  by  NCLB.  That  program  requires  partnerships  to  include  a  state  educational 
agency  or  public  regional  intermediary,  such  as  Pennsylvania’s  IUs,  the  engineering,  math  or 
science  department  of  an  IHE,  and  one  or  more  high-needs  school  districts.  Unlike  the  NSF 
program,  in  which  funds  are  awarded  to  projects  in  a  national  competition,  the  Department  of 
Education  allocates  program  funds  for  states  to  administer. 


Overview  of  the  Math  Science  Partnership  of  Southwest  Pennsylvania 

The  MSP  brings  together  53  K— 12  school  districts — 45  as  part  of  the  NSF  grant  and  eight 
additional  districts  supported  by  a  companion  math  and  science  partnership  grant  from 
the  Pennsylvania  Department  of  Education  (PDE) — four  IUs,  and  four  IHEs  in  Southwestern 
Pennsylvania.  Its  goals  are  to  increase  K-12  students’  knowledge  of  mathematics  and  science, 
increase  the  quality  of  the  K— 16  educator  workforce,  and  create  sustainable  coordination  of 
partnerships  in  the  IUs.  The  project  began  in  September  2003  with  an  initial  funding  period 
of  five  years.  In  2008,  the  NSF  extended  the  end  date  of  the  project  to  2010. 

The  MSP  serves  Southwestern  Pennsylvania,  including  the  urban  fringe  of  the  City  of 
Pittsburgh,  several  smaller  urban  areas,  suburbs,  and  rural  areas.2  Most  of  the  school  districts 
participating  in  the  MSP  are  within  the  regions  served  by  the  four  partner  IUs;  however,  five 
districts  are  in  neighboring  regions  that  are  served  by  other  IUs.  The  school  districts  in  the 
MSP  are  relatively  small:  Total  enrollment  is  approximately  114,000  students,  an  average  of 
about  2,150  per  district.  The  typical  MSP  school  district  has  only  four  or  five  schools.  The  MSP 
districts  have  a  total  of  approximately  3,400  teachers  who  teach  math  or  science  topics.  On 
average,  about  39  percent  of  students  in  MSP  schools  are  economically  disadvantaged,  com¬ 
pared  with  a  statewide  average  of  36  percent.  This  figure  is  higher  in  the  PDE  MSP  districts 
(59  percent)  than  in  the  NSF  MSP  districts  (35  percent).  The  enrollment  of  underrepresented 
minorities  is  approximately  19  percent,  compared  with  a  statewide  average  of  22  percent.3 
Again,  this  figure  is  higher  in  the  PDE  MSP  districts  (25  percent)  than  in  the  NSF  MSP  dis¬ 
tricts  (18  percent).  These  demographics  vary  widely  across  schools.  The  reported  percentages  of 
both  economically  disadvantaged  and  minority  populations  vary  between  0  percent  and  nearly 
100  percent  for  individual  schools. 

At  the  start  of  the  project,  there  was  similar  wide  variation  in  student  achievement  levels 
across  the  MSP  districts.  A  substantial  portion  of  MSP  schools  are  not  making  adequate  yearly 
progress  under  NCLB;  three  MSP  districts  are  subject  to  state  control  if  they  do  not  improve, 
and  one  of  those  is  already  being  operated  under  a  state  board  of  control.  At  the  other  end  of 
the  spectrum,  the  MSP  includes  several  “blue-ribbon”  schools  that  are  among  the  highest- 
achieving  in  the  state.  Chapter  Four  includes  additional  details  about  student  achievement  in 
MSP  schools. 


2  Pittsburgh  Public  Schools,  the  largest  urban  school  district  in  the  region,  is  not  formally  involved  as  an  MSP  participant. 
However,  district  personnel  in  both  mathematics  and  science  have  a  long  history  of  involvement  with  a  regional  collabora¬ 
tive  related  to  the  MSP,  as  discussed  in  Chapter  Two. 

3  Minorities  include  students  identified  as  black,  Hispanic,  Asian,  or  Native  American.  The  great  majority  of  these  are 
black. 
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The  four  partner  IHEs  are  small-  to  midsized,  teaching-oriented,  private  institutions 
located  in  Southwestern  Pennsylvania:  Carlow  University,  Chatham  University,  Robert  Morris 
University,  and  Saint  Vincent  College.  These  IHEs  have  approximately  860  faculty  members 
and  11,300  students.  Approximately  55  members  of  their  math,  science,  engineering,  and  edu¬ 
cation  faculties  are  participating  in  this  project.  Although  some  of  the  larger  research-oriented 
universities  in  the  region  were  invited  to  participate  in  the  MSP,  they  declined.  In  some  cases, 
the  universities  were  already  involved  in  educational  reform  programs.  For  example,  the  Uni¬ 
versity  of  Pittsburgh  School  of  Education  was  already  involved  in  another  Math  and  Science 
Partnership  initiative  through  the  university’s  Learning  Research  and  Development  Center. 


Purpose  of  This  Monograph 

The  purpose  of  this  monograph  is  to  summarize  key  findings  regarding  the  project’s  progress 
toward  meeting  its  three  goals  of  increasing  the  mathematics  and  science  achievement  of  K-12 
students,  increasing  the  quality  of  the  K— 16  educator  workforce,  and  creating  sustainable  part¬ 
nerships.  Analyses  draw  on  data  collected  through  the  MSP’s  first  five  years  of  implementation, 
information  from  surveys  of  teachers  and  principals,  qualitative  data  from  participating  IHEs, 
educator  participation  records,  and  student  achievement  data.  Although  this  monograph  con¬ 
cludes  RAND  Education’s  MSP  evaluation  activities,  it  is  important  to  note  that  it  does  not 
constitute  a  final  evaluation  of  the  MSP.  The  project,  including  data  collection  and  evaluation 
activities,  is  expected  to  continue  through  a  forecasted  end  date  in  2010. 


Organization  of  This  Monograph 

The  remainder  of  this  monograph  is  organized  as  follows.  Chapter  Two  provides  an  overview 
of  the  MSP  intervention  strategies,  motivation,  and  organizational  structure  and  management. 
Chapter  Three  contains  an  overview  of  the  evaluation,  including  a  logic  model  of  the  project, 
the  evaluation  questions,  and  the  design  and  limitations  of  the  evaluation.  Chapters  Four 
through  Six  describe  the  progress  of  the  project  toward  meeting  its  goals  of  improving  student 
achievement,  increasing  the  quality  of  the  educator  workforce,  and  creating  sustainable  part¬ 
nerships,  respectively.  Chapter  Seven  concludes  this  monograph  with  a  summary  of  key  find¬ 
ings.  Three  appendixes  contain  supplemental  details  regarding  analytic  methods  and  results. 
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This  chapter  describes  the  MSP’s  intervention  strategies,  the  motivation  for  its  formation,  and 
its  organizational  and  management  structure.  The  MSP’s  goals  are  consistent  with  the  objec¬ 
tives  of  the  overall  Math  and  Science  Partnership  program.  To  reiterate,  they  are  to  increase 
K-12  students’  knowledge  of  mathematics  and  science,  increase  the  quality  of  the  K— 16  edu¬ 
cator  workforce,  and  create  sustainable  coordination  of  partnerships  in  the  IUs.  The  MSP 
planned  three  crosscutting  intervention  strategies  to  accomplish  these  goals. 


Intervention  Strategies 

The  MSP  intervention  strategies  are  as  follows: 

•  Professional  development  for  content  and  leadership  is  accomplished  through  academies  and 
seminars  for  K-12  educators  and  IHE  faculty.  The  overriding  purpose  of  these  activities 
is  to  equip  teachers  with  the  content,  pedagogy,  and  leadership  skills  necessary  to  become 
effective  leaders  in  their  institutions. 

•  Curriculum  alignment  and  pedagogical  and  course  refinement  are  accomplished  at  the  K-12 
level  through  the  use  of  curriculum  frameworks  and  research-based  curriculum  materials 
and  at  the  IHE  level  through  the  contributions  of  K-12  teachers  who  spend  one  to  two 
semesters  or  a  summer  on  IHE  campuses,  working  with  faculty  to  refine  IHE  courses. 

•  Support  for  and  dissemination  of  research-based  resources  and  tools,  which  is  accomplished 

based  curricula. 

Importantly,  these  intervention  strategies  are  not  distinct  and  separable;  rather,  they  are 
intertwined  in  a  design  that  unites  K-12  and  IHE  educators  in  working  to  achieve  the  three 
primary  goals  of  the  MSP.  Table  2.1  describes  the  primary  activities  used  by  the  MSP  to  enact 
these  strategies. 

Professional  Development  for  Content  and  Leadership 

The  first  intervention  strategy,  professional  development  for  content  and  leadership,  was 
designed  to  create  cadres  of  teacher  leaders  in  partner  K-12  school  districts.  The  teachers  receive 
training  and  then  lead  professional  development  efforts  in  their  districts  through  on-site  acad¬ 
emies  (OSAs).  Each  K-12  district  and  IHE  designates  a  leadership  action  team  (LAT)  to  enact 


5 


6  Math  Science  Partnership  of  Southwest  Pennsylvania:  Measuring  Progress  Toward  Goals 


Table  2.1 

Primary  Activities  of  the  MSP 


Intervention 

Strategy 

MSP  Activity 

Description 

Professional 
development 
for  content  and 
leadership 

Leadership  action 
teams  (LATs) 

Leadership  Action 
Academies 

LATs  represent  each  school  district  and  IHE.  Each  LAT  assesses  the 
strengths  and  weaknesses  of  its  institution  and  develops  an  action 
plan  for  improvement.  The  LATs  select  teachers  and  administrators  to 
participate  in  other  MSP  activities.  District  LATs  meet  four  times  per  year 
at  the  Leadership  Action  Academies,  and  IHE  LATs  meet  as  necessary. 

Teacher  Leadership 
Academies  (TLAs) 

On-site  academies 
(OSAs) 

TLAs  provide  professional  development  on  leadership  and  content  for 
selected  teachers,  grouped  by  discipline  (math  or  science)  and  level  of 
instruction  (early  learners,  elementary,  or  secondary).  The  academies 
meet  for  27  days  over  a  three-year  period.  Participating  teachers  become 
leaders  of  professional  development  in  their  own  districts,  developing 
communities  of  learning  and  holding  OSAs  to  disseminate  what  they 
learn  in  the  TLAs. 

Principal  seminars 

Seminars  for  principals  aim  to  build  a  deeper  understanding  of  effective 
mathematics  instruction  and  to  develop  effective  observing  and 
conferencing  techniques.  These  seminars  total  38  hours  over  a  one-year 
period.  The  MSP  developed  an  additional  module  to  support  science 
education  supervision. 

Content  short 
courses 

Vouchers  and  stipends  support  teachers  who  attend  professional 
development  courses  in  math  or  science  content  areas  sponsored  by  IHE 
partners  and  others,  helping  them  deepen  their  content  understanding. 

Curriculum 
alignment  and 
pedagogical 
and  course 
refinement 

Teacher 
fellows  (TFs) 

The  TF  program  provides  support  for  two  or  more  teachers  from  each 
district  over  the  five-year  grant  period  to  spend  one  or  two  terms  at  a 
partner  IHE.  During  each  term,  the  TF  works  with  IHE  faculty  to  help 
refine  two  IHE  courses,  takes  a  college  course,  and  participates  in  MSP 
activities. 

Curriculum 

frameworks 

The  MSP  developed  a  curriculum  framework  for  science  and  refined  the 
one  for  math,  with  the  six  to  eight  "big  ideas"  to  be  taught  in  these 
disciplines  at  each  K-12  grade  level.  The  frameworks  are  intended  to 
help  make  effective  teaching  of  Pennsylvania's  academic  standards  in 
science  and  math  manageable  by  enabling  teachers  to  focus  their  time 
teaching  fewer  concepts  in  more  depth. 

Support  for  and 
dissemination  of 
research-based 
materials 

Network 

Connections 

conference 

Educator  networks 

This  daylong  conference  is  held  twice  per  year  for  LATs  and  other  math 
and  science  teachers  and  IHE  faculty  to  explore  research-based  resources 
and  tools. 

Networking  activities  assist  districts  in  implementing  challenging  courses 
and  curricula.  Groups  of  teachers  from  across  the  region  (MSP  and  non- 
MSP  districts)  who  are  using  the  same  curricula  (e.g.,  Everyday  Math, 
Connected  Math,  Investigations)  meet  to  share  best  practices.  State- 
funded  math  coaches  are  convened  to  support  shared  learning. 

NOTE:  MSP  activities  are  shown  in  the  table  as  addressing  one  specific  intervention  strategy  on  which  they  are 
likely  to  have  the  greatest  overall  impact.  However,  most  actually  address  additional  intervention  strategies.  For 
example,  the  TF  activity  addresses  not  only  curriculum  alignment  and  pedagogical  and  course  refinement,  but 
also  professional  development  for  content  and  leadership. 


this  strategy.  The  K— 12  LATs  nominally  include  six  teachers  representing  elementary-,  middle-, 
and  high-school  math  and  science,  as  well  as  a  district-level  administrator  and  a  guidance 
counselor;  the  IHE  LATs  nominally  include  faculty  and  department  heads  from  math,  sci¬ 
ence,  and  education  departments.  Each  team  is  charged  with  creating  a  strategic  action  plan  to 
strengthen  the  teaching  and  learning  of  math  and  science  in  its  institution,  particularly  courses 
taken  by  prospective  teachers.  An  important  component  of  the  action  plan  is  to  identify  teacher 
leaders  and  IHE  faculty  who  will  participate  in  Teacher  Leadership  Academies  (TLAs). 
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The  TLAs  and  OSAs  are  expected  to  build  the  capacity  for  change  among  MSP  partners 
by  providing  training  in  math  and  science  content  as  well  as  pedagogy.  These  academies  are 
specialized  by  discipline  (math  or  science)  and  level  of  instruction  (early  learners,  elementary,  or 
secondary)  and  use  previously  developed  professional  development  materials  (described  later). 
Although  the  selected  professional  development  materials  have  not  been  subjected  to  rigorous 
empirical  tests  of  their  efficacy,  they  have  appeared  promising  in  case  studies  during  pilot  test¬ 
ing  (Borko,  2004).  Using  a  two-stage  train-the-trainer  approach,  the  developers  of  the  materi¬ 
als  provided  initial  training  to  MSP  staff,  who  delivered  it  to  teacher  leaders  during  the  TLAs, 
who  then,  in  turn,  delivered  it  to  district  colleagues  during  OSAs.  This  approach  enabled  the 
MSP  to  scale  the  program  to  reach  a  large  number  of  teachers.  It  is  generally  agreed  that 
the  facilitator  plays  a  crucial  role  in  the  success  of  professional  development  programs,  so  this 
strategy  poses  a  potential  challenge:  Is  the  professional  development  ultimately  delivered  effec¬ 
tively  in  the  OSAs?  The  field  has  not  yet  established  whether  such  a  train-the-trainer  approach 
is  effective  (Yoon  et  ah,  2007). 

MSP  professional  development  for  leadership  is  not  limited  to  teachers:  Through  principal 
seminars,  administrators  are  given  an  opportunity  to  gain  a  deeper  understanding  of  effective 
instruction  and  develop  observation  and  conferencing  techniques  to  support  improvements  in 
teacher  instructional  practices.  The  following  paragraphs  describe  several  of  the  resources  used 
in  the  TLAs  and  principal  seminars. 

Video  Cases  for  Mathematics  Professional  Development.  This  series  uses  video  record¬ 
ings  of  instruction  to  facilitate  and  deepen  the  quality  of  educators’  thinking  with  regard  to 
student  learning  of  mathematics.  With  video  of  teachers  leading  whole-class  math  instruc¬ 
tion,  participants  are  encouraged  to  analyze  the  classroom  interactions  (Seago,  Mumme,  and 
Branca,  2003).  The  format  provides  an  opportunity  to  stop,  consider,  and  discuss  learning 
interactions  in  the  moment  in  which  an  event  occurs,  something  that  is  not  possible  during  a 
live  classroom  lesson  (LeFevre,  2003). 

Developing  Mathematical  Ideas.  This  K— 6  series  aids  teachers  in  analyzing  and  compre¬ 
hending  student  learning.  A  facilitator  guides  teachers  through  written  vignettes  of  learning 
events  in  the  classroom,  video  recordings  of  episodes  from  the  vignettes,  participation  in  simi¬ 
lar  math  learning  events,  and  discussion  (Morse  and  Davenport,  2000;  Schifter,  Bastable,  and 
Russell,  1999).  Teachers  are  encouraged  to  journal  about  experiences  in  their  own  classrooms, 
including  moments  of  professional  or  student  struggles,  for  later  discussion  with  colleagues.  In 
its  original  design,  participants  met  weekly  or  biweekly  for  three-hour  sessions  over  eight  weeks 
for  each  of  two  seminars;  the  curriculum  has  variations  for  other  formats,  such  as  summer 
seminars  (Schifter,  Bastable,  and  Russell,  1999).  Teachers  rate  the  seminar  series  favorably  in 
helping  them  adjust  to  a  standards-based  curriculum  and  understand  the  process  of  student 
learning  (Morse  and  Davenport,  2000). 

Implementing  Standards-Based  Mathematics  Instruction.  Implementing Standards-Based 
Mathematics  Instruction  is  a  casebook  for  professional  development  that  provides  a  framework 
offering  educators  a  means  to  evaluate  instructional  decisions,  choices  of  materials,  and  learn¬ 
ing  outcomes,  along  with  case  studies  affording  opportunities  to  ground  these  ideas  in  actual 
classroom  practice  (Smith  et  ah,  2000).  The  casebook  is  a  product  of  a  project  titled  “Quanti¬ 
tative  Understanding:  Amplifying  Student  Achievement  and  Reasoning”  at  the  University  of 
Pittsburgh’s  Learning  Research  and  Development  Center.  The  project  examined  hundreds 
of  middle-school  classroom  lessons  and  drew  findings  regarding  what  constitutes  effective 
mathematics  instruction  (Silver  and  Stein,  1996).  A  U.S.  Department  of  Education  (1997b) 
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white  paper  identified  this  intervention  as  a  promising  practice,  reporting  that  it  helped  build 
capacity  for  improved  mathematics  instruction  and  led  to  increases  in  student  performance. 

Lesson  Study.  Lesson  study  calls  for  a  shift  in  teacher  practice  and  thinking,  requiring 
teachers  to  work  together  to  write,  observe,  revise,  observe  again,  and  analyze  lesson  plans. 
Teachers  observe  the  lesson  being  delivered  in  the  classroom  and  engage  in  professional  con¬ 
versation  focusing  on  how  specific  students  develop  content  understanding.  Following  observa¬ 
tion,  the  teams  revise  the  lesson,  then  a  team  member  implements  the  revised  lesson  while  the 
rest  of  the  team  observes  (Lewis,  Perry,  and  Hurd,  2004).  The  Japanese  credit  the  practice  of 
lesson  study  by  classroom  teachers  as  a  key  factor  in  revolutionizing  their  educational  system 
(Lewis,  2002;  Lewis,  et  ah,  2004;  Takahashi,  2000). 

National  Academy  for  Curriculum  Leadership.  This  program  focuses  on  building  lead¬ 
ership  capacity  and  improving  science  instruction.  Using  the  curriculum  as  a  focal  point,  it 
provides  tools  and  strategies  for  teams  to  improve  their  science  programs  by  selecting  appro¬ 
priate  materials  and  providing  professional  development  to  improve  the  quality  of  instruction. 
St.  John  et  al.  (2006)  found  that  the  program  helped  districts  develop  science  leadership  capac¬ 
ity,  and  participants  reported  that  it  helped  them  improve  the  quality  of  science  instruction  in 
their  districts. 

Lenses  on  Learning.  This  seminar  series  strives  to  shift  thoughts  and  beliefs  away  from 
traditional  approaches  to  leadership  in  schools  and  toward  those  that  are  consistent  with  reform 
efforts  (Stein  and  Nelson,  2003).  In  these  seminars,  administrators  develop  the  skills  to  lead  a 
professional  learning  community  that  values  dialogue  about  student  learning  and  instructional 
change  (Nelson,  1997). 

Although  professional  development  is  based  primarily  on  published  materials,  the  LATs 
are  responsible  for  other  aspects  of  the  MSP  professional  development  requirements,  such  as 
determining  the  goals,  timing,  location,  and  means  of  district  support  for  participants.  The 
LATs  use  district-specific  data  in  developing  these  plans,  including  math  achievement  data 
from  the  Pennsylvania  System  of  School  Assessment  (PSSA),  science  achievement  data  from  an 
assessment  administered  by  the  MSP,1  results  from  a  survey  of  teacher  confidence  in  math  and 
science  content  areas,  results  from  the  District  Profile  on  Course  Completion,  and  an  analysis 
of  strengths,  weaknesses,  opportunities,  and  threats,  which  the  LATs  completed  early  in  the 
first  year.  The  work  of  the  LATs  is  accomplished  over  four  regional  meetings  per  year  at  Leader¬ 
ship  Action  Academies,  and  some  teams  hold  additional  meetings  in  their  districts. 

Curriculum  Alignment  and  Pedagogical  and  Course  Refinement 

The  second  intervention  strategy  of  curriculum  alignment  and  pedagogical  and  course  refine¬ 
ment  is  accomplished  at  the  K-12  level  through  the  use  of  math  and  science  curriculum  frame¬ 
works.  These  curriculum  frameworks  were  developed  by  the  MSP  and  partner  organizations 
in  Southwestern  Pennsylvania.  The  curriculum  frameworks  identify  the  big  ideas  that  should 
be  taught  in  each  discipline,  and  at  each  grade  level,  for  students  to  meet  Pennsylvania’s  aca¬ 
demic  standards  in  math  and  science.  A  primary  goal  is  to  enable  teachers  to  improve  on  their 
prior  practice  by  teaching  fewer  concepts  in  greater  depth,  with  less  repetition  from  year  to 
year.  In  addition,  to  develop  a  plan  for  change  in  the  mathematics  and  science  curricular  area, 
the  LATs  annually  assess  their  districts  using  a  district  development  matrix,  a  tool  based  on 


1  Until  the  2007-08  academic  year,  Pennsylvania  did  not  administer  a  statewide  science  exam.  Chapter  Four  contains 
additional  information  about  the  project-administered  science  assessment. 
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the  Concerns-Based  Adoption  Model  (Loucks-Horsley  and  Stiegelbauer,  1991).  This  tool  indi¬ 
cates  the  stage  of  the  district’s  transition  in  the  adoption  and  implementation  of  challenging 
courses  and  curricula  at  the  elementary-,  middle-,  and  high-school  levels. 

At  the  IHE  level,  course  refinement  is  accomplished  primarily  through  the  MSP  teacher 
fellow  (TF)  program.  Over  the  five-year  course  of  the  project,  the  program  enables  two  or  more 
teachers  from  each  district  to  spend  a  summer,  a  semester,  or  a  full  academic  year  at  a  partner 
IHE.  During  each  term  on  campus,  it  is  expected  that  the  TFs  will  each  work  with  IHE  faculty 
to  help  refine  two  IHE  courses  in  which  preservice  teachers  enroll,  take  a  math  or  science  col¬ 
lege  course,  and  participate  in  MSP  activities.  The  TF  program  is  an  important  MSP  activity 
because  it  addresses  a  number  of  goals:  In  addition  to  course  revision  and  refinement,  partici¬ 
pating  teachers  receive  professional  development  through  their  enrollment  in  college  courses. 
Through  course  revision,  it  is  intended  that  IHE  faculty  will  become  more  familiar  with  state 
and  national  content  standards  and  that  TFs  will  become  more  familiar  with  the  depth  and 
scope  of  specific  content.  Moreover,  the  TF  also  becomes  a  link  between  the  IHE  campus 
and  its  school  or  school  district,  helping  to  meet  the  goal  of  sustainable  partnerships. 

Support  for  and  Dissemination  of  Research-Based  Materials 

The  third  intervention  strategy  of  disseminating  and  supporting  the  use  of  research-based 
resources  and  tools  is  achieved  through  several  activities.  Network  Connections  conferences, 
which  are  held  twice  per  year,  include  resource-partner  fairs  during  which  participants  can 
review  materials  and  speak  with  representatives  from  various  professional  development  and 
curriculum  providers.  Additionally,  educator  networks  bring  together  groups  of  teachers  or 
coaches  from  across  the  region  who  are  using  common  research-based  curricula  so  that  they 
can  support  each  other  and  share  best  practices.  Finally,  two  publications,  the  Math  and  Science 
Collaborative  Journal  and  Math  and  Science  Collaborative  Coordi-net,  are  also  important  ele¬ 
ments  of  the  dissemination  strategy.  They  contain  a  directory  of  mathematics  and  science  pro¬ 
fessional  development  opportunities  in  the  region,  whether  sponsored  by  the  MSP  or  by  other 
organizations,  along  with  reports  of  MSP  activities,  new  developments,  and  lessons  learned. 


Motivation  for  the  MSP 

The  MSP  has  its  origins  in  the  Math  and  Science  Collaborative  (MSC),  a  regional  partner¬ 
ship  among  K-12  school  districts,  IHEs,  and  businesses  that  was  founded  in  Pittsburgh  in 
1994.  With  a  few  exceptions,  school  districts  in  Pennsylvania  are  small  and  independently 
controlled.  The  regional  structure  of  both  the  MSC  and  the  MSP  provides  organizational 
coherence  for  multidistrict  participation  while  recognizing  that  not  every  district  will  choose  to 
join.  The  MSC  embraced  many  of  the  values  that  informed  the  design  of  the  MSP,  including 
a  research-based  philosophy  and  organizational  principles  centered  on  the  use  of  standards  to 
help  guide  instruction  and  promote  student  achievement.  Standards  played  an  important  role 
in  the  MSC,  which  originally  created  district  teams  to  address  new  national  standards  in  math 
and  science.2  In  1997,  the  importance  of  standards  was  reinforced  with  the  publication  of  the 
results  of  the  Trends  in  International  Mathematics  and  Science  Study  (TIMSS).  In  addition, 


2  The  National  Council  of  Teachers  of  Mathematics  produced  the  Curriculum  and  Evaluation  Standards  for  School 
Mathematics  in  1989.  The  National  Research  Council  produced  the  National  Science  Education  Standards  in  1995. 
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the  region  was  one  of  26  jurisdictions  that  participated  in  the  1999  TIMSS  Benchmarking 
Study  (Tananis  et  ah,  2002).  The  TIMSS  results  were  pivotal  to  the  origination  of  the  MSP 
because  they  demonstrated  the  importance  of  standards  in  setting  clear  goals  and  encouraging 
collegial  discussions.  Later,  when  the  Math  and  Science  Partnership  program  was  announced, 
the  MSC  identified  it  as  an  opportunity  to  strategically  apply  what  was  learned  from  TIMSS 
as  a  focus  for  change  in  Southwestern  Pennsylvania. 

Many  of  the  current  MSP  activities,  such  as  Network  Connections  and  the  TLAs,  origi¬ 
nated  in  the  MSC.  The  first  academies  began  in  1995,  featuring  the  national  standards  and  a 
TIMSS  resource  kit  (U.S.  Department  of  Education,  1997a).  The  academies  were  quite  suc¬ 
cessful,  and,  by  1998,  many  districts  in  the  region  had  participated.  These  experiences  with 
the  MSC  informed  the  design  of  the  MSP,  particularly  the  importance  of  leadership,  creating 
district  teams  to  plan  for  change,  providing  adequate  release  time  for  teachers  to  participate  in 
professional  development,  engaging  in  data-driven  decisionmaking,  and  focusing  on  research- 
based  resources. 

Many  of  the  current  partners  were  also  involved  in  the  MSC,  with  slightly  different  roles. 
For  example,  each  of  the  IHEs  was  involved  with  the  MSC  prior  to  the  start  of  the  MSP,  pri¬ 
marily  through  participation  in  meetings  that  showcased  university  programs.  Similarly,  the 
IUs  were  involved  with  the  MSC,  though  their  role  shifted  in  the  MSP  model  to  be  collabora¬ 
tors  and  full  partners  rather  than  more  loosely  associated  providers  of  K-12  professional  devel¬ 
opment  resources  and  materials. 

The  Pittsburgh  Public  Schools  is  another  example  of  a  long-standing  MSC  member  with 
an  ongoing  relationship  with  the  MSP.  The  district  is  not  a  full  partner  in  the  MSP  due  to  other 
commitments;  however,  its  personnel  have  taken  part  in  MSP  activities  that  are  open  to  non- 
MSP  districts.  In  addition,  early  in  the  project,  the  Pittsburgh  Public  Schools  co-hosted  train¬ 
ing  sessions  attended  by  MSP  coordinators.  (The  MSP  coordinator  role  is  described  later.) 


Organizational  Structure  and  Management  of  the  Partnership 

The  organizational  structure  of  the  MSP  consists  of  a  governing  board  known  as  the  MSP 
Cabinet  and  six  project  leadership  teams  focused  on  math,  science,  K-12,  IHEs,  assessment 
and  evaluation,  and  budget  and  finance.  The  project  leadership  teams  have  team  leaders  and 
project  directors  who  guide  the  planning  of  project  activities,  allocate  tasks  among  team  mem¬ 
bers,  and  develop  updates  on  progress  and  challenges  for  the  MSP  Cabinet.  Project  directors 
are  responsible  for  daily  follow-up  on  the  implementation  of  the  team’s  specific  tasks,  ensuring 
that  targets  are  met  on  schedule,  maintaining  project  documentation,  and  providing  progress 
updates. 

The  MSP  Cabinet  is  the  core  decisionmaking  body  and  has  ultimate  responsibility  for 
the  coordination  and  implementation  of  the  partnership,  including  coordination  of  partners 
and  project  leadership  teams.  The  cabinet,  which  meets  monthly,  consists  of  the  MSP  princi¬ 
pal  investigator  (PI),  co-PIs,  and  team  leaders  and  project  directors  from  the  project  leadership 
teams.  District  representatives  are  invited  to  attend  cabinet  meetings,  and  several  have  become 
regular  participants. 

The  math  and  science  leadership  teams  are  responsible  for  strengthening  teaching  and 
learning  in  their  respective  disciplines  at  the  K-12  level.  Each  of  these  teams  includes  an  IHE 
faculty  representative  as  team  leader,  the  MSP  PI,  additional  faculty  representatives  from  the 
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IHEs,  MSP  coordinators,  and  the  math  or  science  project  director.  The  IHE  leadership  team 
is  responsible  for  strengthening  practices  in  the  teaching  and  learning  of  IHE  mathematics 
and  science.  This  team  includes  the  PI,  one  faculty  representative  from  each  of  the  four  IHEs, 
and  the  math  and  science  project  directors.  The  K-12  team  includes  the  PI,  the  K-12  project 
director,  representatives  from  participating  IUs,  and  two  school  district  superintendents.  It  is 
responsible  for  strengthening  and  supporting  district  and  school-wide  understanding  and  sup¬ 
port  for  effective  teaching  and  learning  of  math  and  science.  The  assessment  and  evaluation 
team  is  responsible  for  documenting  student  achievement  and  evaluating  the  project.  The  team 
includes  the  project  PI  and  representatives  from  RAND,  the  University  of  Pittsburgh,  and 
AIU.  The  budget  and  finance  leadership  team  includes  fiscal  representatives  from  all  partner 
institutions  and  subcontractors  and  is  responsible  for  financial  matters.  Each  of  these  teams 
meets  at  least  quarterly,  though  many  have  found  the  need  to  meet  more  frequently  in  order  to 
plan  and  monitor  progress. 

The  MSP  coordinators  are  a  key  component  of  the  organizational  structure.  A  group  of 
math  and  science  educators,  the  coordinators  are  responsible  for  connecting  K-12  districts 
to  IHEs  and  for  implementing  MSP  activities.  The  coordinators  bring  a  wealth  of  experi¬ 
ence  to  the  MSP,  evident  in  their  diverse  backgrounds:  retired,  late-career,  and  early-career 
K-12  teachers;  K-12  administrators;  and  a  community  college  teacher.  The  coordinators  were 
hired  by  AIU  after  demonstrating  a  variety  of  qualifications,  including  math  or  science  con¬ 
tent  knowledge,  oral  communication  skills,  conflict  resolution  skills,  understanding  of  project 
goals,  organizational  planning  skills,  and  motivation.  They  received  25  days  of  in-service  train¬ 
ing  in  the  first  project  year  and  at  least  five  days  in  each  subsequent  year.  They  also  participate 
in  state  and  national  professional  conferences.  Funding  from  the  MSP  grant  supports  the  coor¬ 
dinators,  and  their  work  is  undertaken  primarily  under  the  supervision  of  the  PI. 

In  the  next  chapter,  we  discuss  the  design  of  the  evaluation,  including  the  logic  model  and 
theory  of  action  used  by  the  evaluation  team  as  a  framework  for  its  work. 


CHAPTER  THREE 

Evaluation  Design 


To  evaluate  the  project,  the  MSP  established  an  assessment  and  evaluation  team  that  included 
members  of  three  institutions:  the  Collaborative  for  Evaluation  and  Assessment  Capacity  at  the 
University  of  Pittsburgh,  AIU’s  evaluation  division,  and  the  RAND  Corporation.  Collectively, 
this  team  provided  formative  advice  to  the  project,  presented  interim  Endings  that  gauged  the 
project’s  progress  toward  achieving  its  goals,  and  documented  how  well  the  model  worked  for 
the  benefit  of  future  initiatives  that  may  seek  to  replicate  it.  This  chapter  describes  the  evalua¬ 
tion  design,  including  the  evaluation  questions  and  the  logic  model  and  theory  of  action  that 
guided  the  evaluation.  The  chapter  concludes  with  a  discussion  of  the  data  sources  on  which  we 
drew  for  this  monograph,  a  brief  overview  of  the  analyses  to  be  detailed  in  subsequent  chapters, 
and  the  limitations  of  the  evaluation. 


Evaluation  Questions 

The  evaluation  questions  addressed  here  are  based  on  the  MSP’s  goals: 

•  What  progress  has  been  made  during  the  implementation  of  the  MSP  toward  increas¬ 
ing  K—12  students’  math  and  science  achievement,  and  to  what  extent  can  the  progress  be 
attributed  to  the  MSP? 

•  What  progress  has  been  made  during  the  implementation  of  the  MSP  toward  increasing 
the  quality  of  the  K—16  educator  workforce,  and  to  what  extent  can  the  progress  be  attrib¬ 
uted  to  the  MSP? 

•  What  progress  has  been  made  during  the  implementation  of  the  MSP  toward  creating 
sustainable  partnerships,  and  to  what  extent  can  the  progress  be  attributed  to  the  MSP? 


Logic  Model 

The  evaluation  team  consulted  with  project  staff  to  develop  a  logic  model  to  guide  the  evalu¬ 
ation  (see  Figure  3.1).  Logic  models  are  common  evaluation  tools  that  offer  visual  representa¬ 
tions  of  a  project’s  path  to  achieving  its  intended  outcomes  and  provide  a  unified  set  of  terms 
and  relationships  to  facilitate  discussion  about  the  project.  This  logic  model  includes  the  tradi¬ 
tional  components  of  inputs,  activities,  outputs,  and  outcomes.  In  the  following  sections,  we 
describe  each  component  of  the  logic  model  diagram,  moving  from  left  to  right. 
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Figure  3.1 
MSP  Logic  Model 
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Resources  or  Inputs 

Inputs  are  the  resources  that  support  or  guide  MSP  activities.  These  include  not  only  funding 
and  human  resources,  but  also  materials  and  expertise.  The  NSF  and  the  PDE  provide  the  pri¬ 
mary  funding  to  support  the  MSP  activities.  MSP  staff  facilitate  many  of  the  activities  through 
administrative  tasks  (e.g.,  coordinating  and  maintaining  contact  with  the  various  partners).  As 
described  in  Chapter  Two,  the  LATs  are  the  primary  resource  for  providing  guidance  to  the 
K— 12  school  districts  and  IHEs.  Materials  and  tools,  such  as  student  achievement  data, 
the  district  profile  of  course  completion,  the  district  development  matrix,  and  the  strategic 
action  plans,  are  all  important  in  helping  the  LATs  assess  and  provide  the  appropriate  guid¬ 
ance.  The  project  leadership  teams,  the  MSP  Cabinet,  and  feedback  from  evaluation  pro¬ 
cesses  are  also  key  in  providing  information  that  contributes  to  setting  the  direction  for  MSP 
activities. 

Intervention  Strategies  and  MSP  Activities 

The  inputs  support  the  three  primary  MSP  intervention  strategies  and  the  associated  activi¬ 
ties,  which  were  detailed  in  Chapter  Two.  These  interventions  are  based,  in  part,  on  research 
on  teacher  leadership  development,  teacher  change,  and  communication  strategies  in  learning 
communities  (see,  for  example,  Ball  and  Cohen,  1999;  Loucks-Horsley  and  Stiegelbauer,  1991; 
and  Senge  et  ah,  2000).  The  second  column  of  the  logic  model  in  Figure  3.1  displays  these 
interventions.  Listed  below  each  intervention  are  the  MSP  activities  that  primarily  support  it. 

Outputs 

The  outputs,  in  the  next  column  to  the  right,  are  the  direct  and  tangible  products  of  the  MSP 
activities.  The  major  outputs  from  the  MSP  activities  are  individuals’  exposure  to  skills  and 
knowledge-enriching  activities,  positioning  them  as  instructional  leaders;  materials  and  tools 
that  can  be  accessed  and  utilized  for  courses,  curriculum  development,  and  district  planning; 
and  opportunities  for  interactions  and  networking.  Note  that  there  is  not  an  exact  one-to-one 
correspondence  between  the  MSP  activities  box  and  the  outputs  box.  For  example,  MSP  activi¬ 
ties  listed  under  professional  development  for  content  and  leadership  not  only  produce  instruc¬ 
tional  leaders,  but  also  provide  access  to  inquiry-based  instructional  materials. 

Short-Term  Outcomes 

The  next  three  columns  in  Figure  3.1  list  the  expected  outcomes  that  derive  from  these  out¬ 
puts.  In  the  first  of  these  columns,  the  logic  model  defines  the  short-term  outcomes  that  can 
be  expected  for  the  instructional  leaders,  such  as  increases  in  knowledge,  skills,  and  awareness. 
Midterm  outcomes,  then,  are  the  changes  in  behavior,  policy,  or  practice  that  occur,  presum¬ 
ably,  as  a  result  of  the  increases  in  knowledge,  skills,  and  awareness. 

Midterm  Outcomes 

The  midterm  outcomes  category  of  the  model  is  more  complex  because  it  reflects  the  changes 
that  are  expected  to  occur  at  various  levels.  For  example,  in  the  classrooms,  changes  in  instruc¬ 
tional  practices  are  expected;  at  the  school  level,  changes  in  policies,  such  as  curriculum  align¬ 
ment  and  adoption  of  inquiry-based  instructional  materials,  are  expected;  at  the  district  level, 
expectations  include  improved  administrative  leadership,  implementation  of  on-site  profes¬ 
sional  development  led  by  teacher  leaders,  and  strategic  allocation  of  district  resources;  and 
finally,  at  the  partnership  level,  the  creation  of  professional  learning  communities  for  both 
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K-12  and  IHEs  is  expected.  These  changes  are  not  equivalent  in  terms  of  the  level  of  effort 
required;  changes  in  behavior  may  be  easier  or  more  difficult  to  effect  than  changes  in  policies. 
Moreover,  there  may  be  interdependencies  among  the  changes  listed  as  midterm  outcomes. 
For  example,  improved  administrative  leadership  may  need  to  precede  changes  in  district-  and 
school-level  policies  and  practices.  The  changes  in  classroom  instructional  practices  at  the  IHE 
level  may  need  to  precede  the  implementation  of  challenging  courses. 

Long-Term  Outcomes 

Collectively,  the  changes  described  in  the  midterm  outcomes  column  should  lead  to  the  long¬ 
term  outcomes,  which  are  defined  as  the  MSP’s  three  goals  of  increasing  K-12  students’  knowl¬ 
edge  of  mathematics  and  science,  increasing  the  quality  of  the  K— 16  educator  workforce,  and 
creating  sustainable  coordination  of  partnerships  in  the  IUs. 


Theory  of  Action 

The  theory  of  action  that  underlies  the  MSP  logic  model  is  premised  on  the  view  that  student 
achievement  in  mathematics  and  science  can  be  enhanced  by  administrators  and  classroom 
teachers  who  are  willing  to  become  learners  and  deepen  their  own  conceptual  understanding 
of  the  big  ideas  in  mathematics  and  science  (Shaneyfelt,  2005-2006).  Similar  to  the  theory  of 
action  for  the  NSF-supported  Local  Systemic  Change  Initiative,  this  theory  of  action  argues 
that  providing  teachers  with  opportunities  to  deepen  their  content  and  pedagogical  knowledge 
in  the  context  of  high-quality  instructional  materials  will  result  in  better-prepared  teachers 
(I.  Weiss,  Banilower,  et  ah,  2002).  With  ongoing  support,  teachers  will  be  more  inclined  to 
change  their  instruction  in  ways  advocated  by  national  standards  and  will  have  greater  capacity 
to  do  so.  Improved  instruction  will,  in  turn,  lead  to  higher  student  achievement. 

This  theory  of  action  is  supported  by  a  number  of  research  studies.  In  particular,  Cath¬ 
erine  Lewis’s  seminal  work  on  lesson  study  and  research  by  Japanese  and  U.S.  educators  has 
outlined  key  pathways  to  instructional  improvement,  most  of  which  are  mirrored  in  the  MSP 
logic  model  (Lewis,  2002;  Lewis,  Perry,  and  Hurd,  2004;  Takahashi,  2000).  These  key  path¬ 
ways  include  increased  knowledge  of  mathematics  and  science  subject  matter  and  instructional 
approaches  to  teaching  this  material,  increased  ability  to  observe  students,  stronger  collegial 
networks,  stronger  connection  of  daily  practice  to  long-term  goals,  strong  motivation  and 
sense  of  efficacy,  and  an  improved  quality  of  lesson  plans.  Thus,  the  intervention  strategies  that 
the  MSP  has  employed  provide  mechanisms  for  achieving  program  goals  along  each  of  these 
pathways.  The  TF  program,  content  short  courses,  and  TLAs  are  routes  to  achieving  increased 
knowledge  of  subject  matter.  The  TLAs  also  serve  to  increase  knowledge  of  instruction  and 
leadership.  Stronger  collegial  networks  are  built  via  a  number  of  routes,  including  participation 
in  Network  Connections,  district-led  professional  development  in  OSAs,  and  the  educator  net¬ 
works.  The  emphasis  on  curriculum  alignment  and  pedagogical  and  course  refinement  offers 
opportunities  to  improve  the  quality  of  instructional  materials  and  lesson  plans.  However,  in 
addition  to  these  pathways,  the  MSP  theory  of  action  also  argues  that  support  from  district 
leadership  is  an  important  component  of  instructional  improvement.  Administrators  can  play 
an  important  role  in  supporting  teacher-led  instructional  change  through  supervision  as  well 
as  through  allocation  of  resources,  such  as  time  for  teachers  to  engage  in  professional  develop¬ 
ment.  Thus,  the  principal  seminars  are  a  key  intervention  strategy  for  involving  and  gaining 
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administrator  support.  Finally,  the  role  of  the  IFIE  in  the  MSP  theory  of  action  is  based  on 
the  belief  that  partnerships  between  K-12  districts  and  IHEs  are  mutually  supportive  and  can 
enhance  learning,  cultural  awareness,  and  teaching  practices  for  both  partners. 


Evaluation  Data  and  Analyses 

The  evaluation  data  on  which  we  draw  are  related  to  the  project’s  short-,  mid-  and  long-term 
outcomes.  Though  not  a  focus  of  this  monograph,  it  is  important  to  note  that  the  project  con¬ 
tinues  to  collect  evaluation  data,  tracking  activities,  outputs,  and  outcomes.  Partners  on  the 
assessment  and  evaluation  team  are  collecting  data  from  case  studies  of  K-12  school  districts, 
observations  of  MSP  events,  interviews  with  key  project  personnel,  and  assessments  of  teacher 
learning  during  professional  development.  When  completed,  analyses  of  those  data  may  help 
to  explain  or  interpret  the  findings  reported  herein. 

Each  of  the  following  three  chapters  describes  analyses  and  findings  related  to  the  three 
evaluation  questions.  These  analyses  chiefly  rely  on  the  following  sources  of  data: 

•  Mathematics  and  science  achievement  data  for  K-12  students :  These  data  include  mathe¬ 
matics  scores  from  the  2000-01  through  the  2006-07  academic  years  and  science  scores 
from  2003-04  through  2006-07.  Student  achievement  data  are  described  in  more  detail 
in  Chapter  Four. 

•  Survey  data  from  K—12  educators:  Baseline  and  follow-up  surveys  of  principals  were 
administered  in  2004  and  2006;  the  principal  of  each  MSP  school  was  included  in  the 
sample.  Baseline  and  follow-up  surveys  of  teachers  were  administered  in  2004  and  2007; 
a  stratified  sample  of  teachers  of  mathematics  or  science  were  surveyed.  The  surveys  are 
described  in  more  detail  in  Chapter  Five. 

•  IHE  qualitative  data:  IFIE  qualitative  data  include  semistructured  interviews  with 
56  IFIE-related  faculty  members  and  administrators,  including  deans,  department  admin¬ 
istrators,  and  student-teacher  placement  coordinators;  10  teacher  fellows  (at  least  two  from 
each  IFIE);  four  student  teachers;  and  four  K-12  teachers  who  supervised  student  teach¬ 
ers.  Some  of  these  participants  were  interviewed  in  two  or  more  years.  In  sum,  a  total 
of  118  interviews  were  conducted.  In  addition  to  the  individual  interviews,  three  focus 
groups  were  held  with  teachers  who  had  returned  to  K-12  instruction  after  participating 
in  the  TF  program.  The  evaluation  team  also  observed  one  math  or  science  classroom  at 
each  of  the  IFIEs  and  reviewed  such  documents  as  IFIE  course  syllabi  and  curricula,  ques¬ 
tionnaires  regarding  course  revision,  reports  prepared  by  participants,  and  meeting  notes. 
These  data  were  collected  during  the  2003-04  through  2007-08  academic  years. 

In  addition  to  these  main  sources,  our  analysis  made  use  of  data  from  a  database  of  edu¬ 
cator  participation  in  MSP  activities;  a  statewide  database  on  school  district  demographics, 
finances,  and  achievement;  and  the  project’s  entries  in  an  NSF-sponsored  database  that  collects 
information  annually  on  all  math  and  science  partnership  projects. 

The  analytic  approaches  to  addressing  the  evaluation  questions,  presented  in  detail  in  the 
following  chapters,  can  be  summarized  as  follows: 
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•  Chapter  Four  examines  changes  in  K-12  student  mathematics  and  science  test  scores  and 
uses  three  statistical  approaches  to  examine  the  relationship  of  those  changes  to  educator 
participation  in  the  project. 

•  Chapter  Five  examines  changes  in  midterm  outcomes  associated  with  the  quality  of  the 
K-16  educator  workforce.  The  chapter  includes  both  qualitative  analyses  of  IHE  data 
and  statistical  analyses  of  survey  data  to  examine  the  relationship  of  changes  to  educa¬ 
tor  participation  in  the  project.  Indicators  include  increased  awareness  and  knowledge  of 
research-based  instructional  practices  and  materials,  changes  in  classroom  instructional 
practices,  changes  in  institutional  policies  and  practices,  adoption  of  inquiry-based  mate¬ 
rials,  and  improved  leadership. 

•  Chapter  Six  examines  sustainable  partnerships  using  indicators  drawn  from  qualitative 
analysis  of  IHE  data,  such  as  the  development  of  the  partnership,  implementation  of 
challenging  courses  through  mechanisms  established  by  the  MSP,  and  changes  in  institu¬ 
tional  policies  and  practices  that  appear  to  foster  or  hinder  sustainability. 


Limitations  of  This  Evaluation 

As  is  commonly  the  case,  this  evaluation  was  selective  in  its  data  collection  and  analyses,  pri¬ 
marily  assessing  the  project’s  achievement  of  its  goals  and  the  major  pathways  toward  achieving 
those  goals.  In  addition,  this  study  was  not  implemented  as  a  randomized  experiment  with  a 
control  group,  and  this  fact  limits  the  ability  to  make  definitive  causal  claims  about  the  MSP’s 
impact.  The  sparse  availability  of  student  achievement  scores  prior  to  the  start  of  the  proj¬ 
ect,  particularly  at  the  lower  grade  levels,  hinders  the  ability  to  assess  the  project’s  impact  on 
achievement  in  elementary  schools.  Another  important  consideration  is  the  lack  of  a  statewide 
science  test  in  Pennsylvania.  Although  the  project  administered  a  science  exam  to  students  in 
participating  districts,  the  exam  was  not  administered  widely  outside  the  project,  so  it  was  not 
possible  to  compare  the  science  performance  of  MSP  districts  to  external  references,  such  as 
comparison  districts.  Until  recently,  the  state  administered  a  math  test  to  students  in  only  a 
few  grade  levels,  so  scores  from  immediately  prior  to  the  start  of  the  MSP  are  not  uniformly 
available  for  all  students  to  serve  as  baselines  for  math  achievement.  Lags  in  the  reporting 
of  achievement  data  forced  the  evaluation  to  confine  its  analyses  to  test  results  through  the 
2006-07  academic  year,  covering  only  four  full  years  of  MSP  implementation  in  the  school 
districts.  If  the  MSP  intervention  strategies  require  more  than  four  years  to  affect  achievement 
outcomes,  the  evaluation  may  not  detect  this  impact.  Finally,  sparse  data  linking  students  to 
teachers  limit  the  numbers  of  students,  teachers,  schools,  and  districts  included  in  student- 
level  achievement  models.  To  supplement  and  illuminate  the  student  achievement  analyses, 
the  evaluation  relies  heavily  on  self-report  data  from  surveys,  interviews,  and  focus  groups  of 
a  sample  of  MSP  participants.  These  data  are  subject  to  common  potential  biases  associated 
with  such  data. 


CHAPTER  FOUR 


Student  Achievement 


This  chapter  presents  analyses  and  findings  related  to  the  first  evaluation  question:  What  prog¬ 
ress  has  been  made  during  the  implementation  of  the  MSP  toward  increasing  K-12  students’ 
math  and  science  achievement,  and  to  what  extent  can  the  progress  be  attributed  to  the  MSP? 
As  specified  in  the  MSP  logic  model  and  theory  of  action,  the  participation  of  educators  in 
MSP  activities  is  a  crucial  first  step  in  the  pathway  to  long-term  outcomes,  such  as  increases 
in  student  achievement.  This  chapter  begins  by  describing  participation  information  collected 
by  the  MSP  and  how  it  was  used  to  construct  participation  measures  for  use  in  statistical  analy¬ 
ses  of  student  achievement  in  this  chapter  and  of  survey  responses  in  Chapter  Five.  Next,  the 
various  types  of  student  achievement  data  available  for  this  evaluation  are  described.  Finally, 
the  chapter  examines  participation  and  achievement  trends,  as  well  as  the  relationship  between 
educator  participation  in  the  MSP  and  student  achievement  outcomes. 


Measures  of  MSP  Participation 

This  section  describes  the  database  of  participation  maintained  by  the  MSP  and  how  it  was  used 
to  construct  annual  and  cumulative  participation  measures  for  both  individuals  and  districts. 

Participation  Database 

The  MSP  project  maintains  comprehensive  information  on  the  level  of  participation  by  every 
educator  in  MSP  school  districts  who  is  eligible  to  participate  in  project  activities,  as  well  as 
on  other  participants  who  are  members  of  non-MSP  school  districts,  IFIEs,  or  other  organiza¬ 
tions.  The  project  database  includes  the  following  information  about  each  participant  through¬ 
out  the  duration  of  the  project: 

•  the  number  of  hours  in  which  the  individual  engaged  in  each  MSP  activity 

•  the  dates  of  those  activities 

•  institutional  affiliation  (e.g.,  district,  school,  IHE) 

•  job  function  (e.g.,  teacher,  principal,  superintendent,  guidance  counselor,  professor) 

•  subject  areas  taught  (e.g.,  math,  science,  special  education) 

•  years  of  experience 

•  demographic  information,  such  as  race/ethnicity  and  gender. 
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Development  of  Participation  Measures  for  Analysis 

The  evaluation  team  used  information  from  this  database  spanning  the  first  four  years  of  MSP 
implementation1  to  develop  individual-  and  district-level  participation  variables  for  use  in  the 
analyses.  These  measures  of  participation  summarize  the  time  spent  by  educators  participating 
in  math  or  science  activities  that  were  identified  by  MSP  staff  as  most  likely  to  be  strongly  asso¬ 
ciated  with  changes  in  classroom  teaching  and  student  learning.  Those  activities  are  as  follows: 
content  short  courses,  educator  networks,  TLAs,  principal  seminars,  and  OSAs.  The  measure 
of  individual-level  participation  is  defined  as  the  total  number  of  hours  that  educators  spent 
engaging  in  these  five  MSP  activities.  Excluded  from  this  calculation  are  activities  whose  influ¬ 
ence  over  classroom  teaching  is  less  direct  or  potent,  including  the  leadership  action  teams  and 
academies,  the  TF  program,  curriculum  framework  development,  and  Network  Connections 
conference  participation. 

In  addition  to  the  individual-level  participation  measure,  three  district-level  participation 
measures  were  developed  to  summarize  the  time  spent  participating  in  MSP  activities  by  dis¬ 
trict  leaders,  mathematics  teachers,  and  science  teachers.  Participation  by  leaders  and  participa¬ 
tion  by  teachers  are  treated  separately  because  these  types  of  educators  have  different  levels  of 
potential  influence  over  changes  in  classroom  instruction  and  student  learning.  Educators  are 
included  in  the  leadership  participation  measure  if  they  have  roles  that  might  enable  them  to 
directly  influence  more  than  one  teacher  in  the  district.  These  include  superintendents,  prin¬ 
cipals,  teacher  leaders,  coordinators,  and  coaches.  Additionally,  educators  who  participated 
in  Leadership  Action  Academies,  TLAs,  or  educator  networks  for  coaches  were  counted  among 
leaders  for  this  participation  measure  because  these  activities  are  intended  to  foster  teacher 
leadership  and  coaching  roles  that  extend  beyond  the  participants’  own  classrooms.  Therefore, 
the  district  leadership  participation  measure  is  defined  as  the  total  hours  of  participation  by 
leaders  in  the  five  activities  listed  earlier.2  This  sum  is  sensitive  to  both  the  number  of  leaders 
who  participated  and  the  amount  of  time  that  each  devoted  to  MSP  activities. 

Each  district’s  remaining  MSP  participants  (those  not  classified  as  leaders)  were  included 
in  the  teacher  participation  measures,  which  are  calculated  separately  for  math  and  science. 
The  calculation  begins  by  summing  the  total  participation  hours  of  full-time-equivalent  teach¬ 
ers,  student  teachers,  substitute  teachers,  teaching  aides,  administrators  (other  than  superinten¬ 
dents  and  principals),  counselors,  and  librarians  in  the  district.3  This  value  is  then  adjusted  to 
create  a  measure  that  is  not  overly  sensitive  to  district  size,  but  instead  captures  each  district’s 
uptake  of  the  participation  opportunities  afforded  by  the  MSP.  The  amount  of  opportunity 
to  participate  varies  by  district  according  to  the  number  of  schools  and  their  configuration, 
but  not  school  size.  The  project  specifies  a  maximum  number  of  participants  per  school  that 
depends  on  school  level:  60  for  each  elementary  school,  20  for  each  middle  school,  and  40  for 
each  high  school.  Thus,  the  district-level  adjusted  teacher  participation  is  calculated  as  the  total 


1  The  participation  measures  include  only  four  years  of  participation  data  because  the  latest  available  student  achievement 
scores  are  from  the  end  of  the  fourth  year. 

2  For  the  survey  analyses,  similar  leadership  participation  measures  were  calculated  at  the  school  level,  as  described  in 
Chapter  Five. 

3  The  vast  majority  of  educators  included  in  the  teacher  measures  were  identified  as  classroom  teachers  by  the  participation 
database;  however,  in  a  few  cases,  other  roles,  such  as  counselor  or  librarian,  were  indicated. 
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district-level  participation  hours  by  teachers  divided  by  the  maximum  number  of  teachers  in 
a  district  who  could  have  possibly  participated  in  MSP  training.4 

Limitations  of  the  Participation  Measures 

Collectively,  these  measures  enable  examination  of  the  relationship  between  MSP  participa¬ 
tion  and  student  achievement  with  available  data.  However,  it  is  important  to  recognize  that 
hours  of  participation  is  an  imperfect  measure  of  all  aspects  of  participation.  In  particular,  the 
measure  does  not  incorporate  information  about  the  quality  of  the  professional  development 
activities  or  the  engagement  of  educators  in  these  activities,  nor  does  it  assess  whether  teach¬ 
ers  successfully  enact  in  their  classrooms  what  they  learn  in  the  MSP  training.  Composites  of 
these  factors  might  produce  better  measures  for  examining  the  relationship  between  MSP  par¬ 
ticipation  and  student  achievement.  The  evaluation  team  explored  the  development  of  a  valid 
and  reliable  composite  measure  that  would  consider  participant  engagement  along  with  hours 
of  participation.  This  effort,  though  promising,  was  not  completed  in  time  to  be  included  in 
this  monograph. 


Measures  of  Student  Achievement 

This  section  describes  the  student  achievement  measures  available  for  mathematics  and  science 
and  the  database  of  test  scores  constructed  by  the  assessment  and  evaluation  team. 

Mathematics  Assessments 

Student  performance  on  the  PSSA  was  used  to  measure  mathematics  achievement.  The  PSSA 
is  a  standards-based,  criterion-referenced  assessment  used  to  measure  students’  attainment  of 
the  state’s  academic  standards.  The  PSSA  uses  four  performance-level  descriptors:  the  advanced 
level  reflects  superior  academic  performance,  proficient  reflects  satisfactory  academic  perfor¬ 
mance,  basic  reflects  marginal  academic  performance,  and  below  basic  reflects  inadequate  aca¬ 
demic  performance.  These  performance  levels  are  reported  publicly  for  all  schools  in  the  state, 
and  the  evaluation  team  is  collecting  this  information.  In  addition,  all  MSP-participating 
school  districts  are  requested  to  provide  student-level  PSSA  results.  Some  districts  have  pro¬ 
vided  this  information  dating  as  far  back  as  the  2000-01  school  year.  The  evaluation  team 


4  The  following  formulas  are  used  to  calculate  the  district-level  adjusted  teacher  participation  variables: 

District-level  adjusted  teacher  =  total  participation  ^ ^  +  total  participation^  ^  +  total  participation^  ^ 
participation  in  mathematics  total  district  participation  slots  available  in  math 

District-level  adjusted  teacher  =  total  participation  +  total  participation  ^  _  +  total  participation^  ^ 

participation  in  science  total  district  participation  slots  available  in  science 

Because  elementary  teachers  usually  do  not  specialize  in  a  particular  subject  area,  participation  in  math  or  science  activities 
might  influence  their  teaching  of  both  subjects.  Therefore,  the  sum  of  elementary  teachers’  participation  regardless  of  sub¬ 
ject  area  is  used  in  the  calculation  of  district-level  teachers’  total  participation  for  both  subject  areas.  Although  this  results  in 
these  teachers’  participation  being  counted  for  both  the  math  and  science  district-level  measures,  only  one  of  the  measures 
is  used  at  a  time  in  statistical  models,  so  participation  is  not  double-counted  in  the  analyses. 
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used  available  student-level  PSSA  results  from  the  2000-01  to  2006-07  school  years  in  the 
analyses  to  assess  math  achievement. 

Science  Assessments 

Until  2008,  the  PDE  did  not  administer  a  statewide  science  exam;  at  that  time,  one  was  ini¬ 
tiated  for  grades  4,  8,  and  11.  Until  this  exam  appeared,  the  project  annually  administered  a 
science  assessment  based  on  the  1995  TIMSS  that  was  provided  by  Promoting  Rigorous  Out¬ 
comes  in  Mathematics  and  Science  Education  (PROM/SE),  a  math  and  science  partnership  at 
Michigan  State  University  (see  Michigan  State  University,  undated).  The  MSP  has  no  access  to 
science  achievement  data  prior  to  the  start  of  the  project,  nor  does  it  have  such  data  from  other, 
non-MSP  districts  in  the  state.  Moreover,  the  PROM/SE  assessment  uses  matrix  sampling, 
which  does  not  produce  valid  student-level  scores.5  Therefore,  results  must  be  aggregated  across 
groups  of  students  to  produce  valid  and  reliable  measures.  These  results  were  aggregated  to  the 
district  level  for  use  in  the  district-level  analysis  described  next.6 

Student  Test  Score  Database 

The  assessment  and  evaluation  team  created  a  database  of  student-level  test  scores  from  MSP 
districts.  Through  the  2006-07  academic  year,  the  database  contained  more  than  340,000 
math  and  science  scores.  Table  4.1  depicts  the  arrangement  of  the  number  of  student  scores  by 
subject  area,  year,  and  student  cohort.  Table  rows  represent  years,  and  columns  represent  stu¬ 
dent  grade  levels.  Shaded  rows  indicate  years  prior  to  implementation  of  the  MSP.  The  quan¬ 
tities  in  the  table  cells  indicate  the  number  of  student  math  and  science  scores  available  for  a 
particular  cohort  of  students  in  a  particular  year.  Math-score  counts  are  indicated  by  the  letter 
M,  and  science-score  counts  by  the  letter  S.  Diagonals  represent  cohorts  of  students.  For  exam¬ 
ple,  the  diagonal  representing  cohort  G  shows  that  the  database  contains  about  9,700  scores 
from  when  these  students  took  the  8th-grade  PSSA  math  exam  in  2003-04  and  about  9,300 
scores  from  they  took  the  llth-grade  PSSA  math  exam  in  2006—07.  In  general,  for  cohorts  of 
students  with  pre-  and  post-tests  separated  by  two  or  three  years,  the  database  contains  scores 
at  both  time  points  for  about  two-thirds  of  the  students. 

In  addition,  the  evaluation  team  requested  detailed  information  about  the  math  and 
science  classes  in  which  students  were  enrolled  and  who  taught  those  classes.  These  student- 
teacher  links  were  used  to  examine  the  association  between  individual  students’  achievement 
and  those  students’  exposure  to  educators  who  participated  in  MSP  activities.  However,  not 
all  districts  provided  student-teacher  link  data,  and  those  that  did  often  provided  incomplete 
information.  Due  to  the  limited  number  of  students  for  whom  student-teacher  links  were  avail¬ 
able,  this  analysis  was  feasible  for  only  four  cohorts  of  students.  The  numbers  of  students  with 
complete  data  in  those  cohorts  ranged  from  approximately  1,840  to  4,100. 


5  In  matrix  sampling  assessment  designs,  students  receive  different  sets  of  items.  This  enables  broader  domain  coverage 
than  when  all  students  receive  the  same  items. 

6  Aggregation  at  the  school  level  was  not  possible  because,  in  some  schools,  there  was  an  insufficient  number  of  students 
tested  to  produce  valid  scores. 
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Table  4.1 

Math  and  Science  Achievement  Scores,  by  Year  and  Cohort 


Grade 

Year 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

2000-01 

3,847  M 

3,941  M 

3,441  M 

2001-02 

6,001  M 

5,584  M 

5,500  M 

Pre-MSP 

2002-03 

6,054  M 

6,261  M 

5,602  M 

scores 

2003-04 

8,261  S 

9,145  M 

8,933  S 

9,696  M 

9,059  M 

2004-05 

7,648  M 

7,873  S 

8,985  M 

8,700  S 

9,914  M 

8,115  S 

9,005  M 

2005-06 

8,870  M 

9,429  M 
9,091  S 

9,595  M 

10,199  M 

10,631  M 
9,516  S 

10,852  M 

9,176  S 

10,324  M 

2006-07 

7,800  M 

8,051  M 
7,664  S 

8,051  M 

8,541  M 

9,137  M 
8,448  S 

9,086  M 

8,273  S 

9,298  M 

Cohort  N 

Cohort  M 

Cohort  L 

Cohort  K 

Cohort  J 

Cohort  1 

Cohort  H 

Cohort  G 

Cohort  F 

NOTE:  Table  rows  represent  years,  and  columns  represent  student  grade  levels.  Shaded  years  were  prior  to  the 
implementation  of  the  MSP.  Quantities  in  cells  indicate  the  number  of  student  math  and  science  scores  available 
for  each  year  and  grade  level.  Math  score  counts  are  indicated  by  the  letter  M,  and  science  score  counts  by  the 
letter  S.  Diagonals  represent  cohorts  of  students. 


Analytic  Approaches  to  Relating  Student  Achievement  to  MSP  Participation 

The  evaluation  team  has  taken  a  variety  of  approaches  to  analyzing  the  relationship  between 
MSP  participation  and  changes  in  student  achievement:  (1)  student-level  achievement  models 
with  linked  educator  participation,  (2)  district-level  achievement  models  with  district-wide 
leader  and  teacher  participation,  and  (3)  statewide  comparison-group  achievement  models.  The 
first  two  achievement  models  examine  pre-post  achievement  scores  within  cohorts  of  students 
in  MSP  schools  and  attempt  to  control  for  the  influence  of  other  factors  on  student  achieve¬ 
ment.  The  statewide  analysis  attempts  to  isolate  the  estimates  of  achievement  change  from 
statewide  trends  in  test  scores.  Despite  these  careful  efforts  to  control  for  potential  biases,  it  is 
important  to  note  that  these  analyses  do  not  support  causal  inferences  about  the  relationship 
between  MSP  participation  and  achievement. 

Student-Level  Achievement  Models 

In  the  student-level  achievement  analyses,  the  evaluation  team  examined  the  association 
between  individual  students’  achievement  change  and  their  linked  educators’  MSP  participa¬ 
tion.  These  analyses  were  possible  for  only  a  fraction  of  students  and  teachers,  so  caution  is 
warranted  in  interpreting  the  results.  Due  to  the  limited  availability  of  data  linking  students 
to  teachers  across  years,  the  proportion  of  students  who  contributed  to  the  analyses  ranged 
from  20  to  45  percent  of  the  students  in  the  analyzed  cohorts,  and  the  proportion  of  teachers 
ranged  from  5  to  15  percent  of  teachers  in  MSP  districts.  However,  the  included  teachers  are 
similar  to  the  larger  sample  in  terms  of  their  average  level  of  participation.  The  proportion  of 
districts  included  in  these  analyses  ranged  from  25  to  57  percent,  and  the  percentage  of  schools 
included  ranged  from  27  to  70  percent.  The  represented  schools  and  districts  span  the  range 
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of  school-  and  district-level  participation  values;  this  information  is  presented  graphically  in 
Appendix  A. 

For  each  cohort  analyzed,  a  multilevel,  mixed-effect,  cross-classified  model  was  used  to 
predict  the  post-test  scores,  using  pretest  scores,  demographic  variables  (race/ethnicity,  gender, 
and  socioeconomic  status),  and  individual-level  participation  and  teaching  experience  of  linked 
educators  in  each  year  as  covariates,  accounting  for  the  clustering  of  students  in  classrooms. 
Additional  details  about  the  models  and  the  variables  included  in  the  analyses  are  presented  in 
Appendix  A. 

District-Level  Achievement  Models 

As  mentioned,  student-level  analyses  were  able  to  examine  only  a  subset  of  students  and  teach¬ 
ers  due  to  the  absence  of  student-teacher  linkages  for  many  students.  To  conduct  analyses  that 
are  inclusive  of  a  greater  proportion  of  students,  the  evaluation  team  also  considered  aggregate 
analyses  at  either  the  school  or  district  level.  However,  the  team  determined  that  the  available 
data  did  not  support  school-level  analyses.  For  most  cohorts,  analyses  spanned  grade  levels 
in  which  students  typically  transition  between  elementary  and  middle  schools  or  middle  and 
high  schools.  Attempting  to  properly  account  for  the  influence  of  educators  on  students  who 
spent  time  in  two  or  more  schools  would  have  greatly  complicated  the  analyses.  On  the  other 
hand,  restricting  the  analyses  to  students  who  remained  in  a  single  school  would  have  excluded 
more  than  half  of  the  schools  from  the  analyses.  Moreover,  the  school-level  analysis  would  have 
been  vulnerable  to  measurement  error.  For  math,  the  number  of  available  achievement  scores 
at  the  school  level  varied  from  one  to  more  than  600.  For  science,  15  different  forms  of  the 
PROM/SE  science  achievement  test  were  administered  at  each  school,  and  the  number  of  stu¬ 
dents  taking  each  form  of  the  test  ranged  from  two  to  40  per  school.  As  a  result  of  these  factors, 
the  evaluation  team  focused  on  conducting  district-level  achievement  analyses. 

The  analyses  examine  the  association  between  district-level  achievement  change  and 
district-level  educator  participation.  All  students  in  the  analyzed  cohorts  for  whom  pre- 
and  post-MSP  test  scores  were  available  were  included  in  the  analysis.  Where  few  of  the  district’s 
math  teachers  participated  in  the  MSP,  any  impact  of  those  participants  would  be  diluted  by 
inclusion  of  students  of  nonparticipating  teachers.  However,  the  MSP  is  a  district-wide  inter¬ 
vention  expected  to  contribute  to  the  improvement  of  mathematics  and  science  achievement 
among  all  students.  In  these  analyses,  a  linear  regression  model  was  used  to  predict  district- 
level  aggregated  math  achievement  according  to  several  factors:  prior  district-level  aggregated 
math  and  reading  achievement;  the  ratios  of  female,  minority,  and  economically  disadvan¬ 
taged  students;  district-level  leaders’  total  participation;  and  adjusted  teachers’  participation. 
As  leaders’  participation  in  math  and  science  might  influence  teaching  and  learning  in  both 
subject  areas,  the  sum  of  leaders’  participation  hours  in  math  and  science  was  included  in  the 
regression  model.  Moreover,  the  adjusted  teachers’  participation  in  math  or  science  was  used 
in  the  corresponding  regression  models.  Additional  details  about  the  models  and  the  variables 
included  in  the  analyses  are  presented  in  Appendix  A. 

Statewide  Comparison  Group  Achievement  Models 

Because  the  MSP  project  design  did  not  include  a  control  group,  the  evaluation  team  used  an 
alternative  method  to  form  comparison  groups  of  school  districts.  The  primary  objective  in 
forming  such  comparison  groups  was  to  make  them  as  equivalent  as  possible  to  the  treatment 
group  (the  MSP  districts)  at  the  start  of  the  project.  In  cases  in  which  student  achievement 
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analysis  is  planned,  equivalence  on  achievement  and  other  variables  associated  with  achieve¬ 
ment  is  desired.  To  the  extent  that  equivalence  is  attained,  subsequent  achievement  differ¬ 
ences  between  the  groups  might  plausibly  be  attributed  to  the  treatment  (participation  in  the 
MSP).  However,  there  is  a  risk  that,  even  if  the  groups  appear  to  be  equivalent  on  the  observed 
variables,  there  may  be  preexisting  differences  between  the  groups  that  affect  achievement. 
Thus,  this  analysis  is  not  as  rigorous  as  would  be  afforded  by  an  experimental  design,  in  which 
randomization  helps  ensure  that  the  groups  are  equivalent  on  both  observed  and  unobserved 
variables. 

The  comparison  groups  were  formed  from  other  school  districts  not  participating  in  the 
MSP.  Pennsylvania  school  districts  were  selected  because  state- controlled  factors,  such  as  test¬ 
ing  and  accountability,  are  the  same  as  for  the  MSP  districts,  and  because  achievement  results 
are  publicly  available  for  all  of  the  districts.  The  evaluation  team  utilized  an  extensive  set  of 
variables,  measured  prior  to  the  start  of  the  project,  in  a  method  known  as  propensity  weighting 
(Hirano,  Imbens,  and  Ridder,  2003;  Rosenbaum,  1987).  The  variables,  assembled  by  RAND 
for  a  study  by  McCaffrey  and  Hamilton  (2007),  included 

•  student  achievement:  mathematics  and  reading  PSSA  achievement  test  scale  scores  for 
fifth-,  eighth-,  and  llth-grade  students  from  1998  to  2003  and  trends  of  change  in  those 
scores 

•  student  demographics:  the  percentages  of  students  in  racial/ethnic  groups  and  low-income 
families 

•  district  finance  and  staffing:  enrollment,  attendance,  graduation  rates,  number  of  teachers, 
pupil-teacher  ratios,  teacher  years  of  experience,  teacher  salaries,  instructional  expenses, 
taxable  property  values,  and  local  tax  revenues 

•  2000  census  measures:  employment,  education,  and  income  of  the  population  living  in  each 
school  district;  property  values;  rents;  and  the  proportion  of  female-headed  households. 

A  first  comparison  group  included  all  non-MSP  school  districts  in  Pennsylvania  except 
the  Pittsburgh  and  Philadelphia  city  school  districts  (because  their  characteristics  as  large 
urban  districts  make  them  very  different  from  the  MSP  districts)  and  one  district  that  did  not 
have  any  enrolled  students.  A  second  comparison  group  was  similar  to  the  first,  except  that 
school  districts  participating  in  the  Philadelphia  Math  and  Science  Partnership  project  were 
also  excluded  from  the  comparison  group.  Finally,  a  third  comparison  group  further  excluded 
districts  in  Southwestern  Pennsylvania  that  were  not  part  of  the  MSP  project  but  nonetheless 
participated  in  some  MSP  activities.  After  creating  the  groups,  balance  was  assessed  on  all  of 
the  variables  used  in  matching.  Most  balanced  well,  and  those  that  did  not  were  included  as 
covariates  in  subsequent  statistical  models.  Appendix  A  includes  tables  of  the  covariates  and 
their  balance. 

Because  no  statewide  science  exam  is  administered  and  the  comparison  districts  did  not 
administer  the  PROM/SE  science  achievement  test,  this  comparative  analysis  is  confined  to 
math  achievement.  The  PDE  discontinued  publicly  reporting  scaled  scores  in  2004,  so  profi¬ 
ciency  levels  were  used  in  models  that  assess  changes  in  achievement  since  the  project  began. 
Specifically,  this  analysis  uses  the  percentage  of  students  scoring  advanced  or  proficient  on  the 
PSSA.  Separate  models  were  run  for  each  cohort  of  students  for  which  PSSA  mathematics  pro¬ 
ficiency  levels  were  available  for  both  2003-04  or  earlier  (as  a  pre-MSP  measure)  and  2006-07 
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(as  a  post-MSP  measure).  Additional  details  about  the  models  and  the  variables  included  in  the 
analyses  are  presented  in  Appendix  A. 


Descriptive  Summary  of  Participation  by  K-12  Teachers  and  Leaders 

Over  the  course  of  the  first  four  years  of  the  project,  58  percent  of  mathematics  and  science 
educators  in  the  MSP  districts  participated  in  project  activities.  Overall,  3,568  educators  par¬ 
ticipated  in  math-related  activities,  and  1,321  participated  in  science-related  activities,  though 
science  participation  was  increasing  in  the  later  years.  About  6  percent  of  elementary-school 
teachers  participated  in  both  math  and  science  activities.  Figure  4.1  displays  the  trends  in  the 
number  of  educators  participating  each  year  by  subject  area.  The  difference  between  math 
and  science  may  be  due  to  ongoing  NCLB -related  pressures  to  improve  mathematics  achieve¬ 
ment,  along  with  the  phase-in  of  lower-stakes  state  science  assessments  over  the  course  of  the 
project. 

Figure  4.2  shows  individual-level  participation  in  math  and  science  across  the  first  four 
years  of  the  project.  For  both  subject  areas,  average  individual-level  participation  started  low 
in  the  first  year,  reached  a  peak  in  the  second  year,  and  diminished  over  the  third  and  fourth 
years.  On  average,  the  total  number  of  hours  that  each  participant  spent  engaging  in  MSP 
activities  over  the  four  years  was  similar  for  the  two  subject  areas.  The  mean  total  participation 
for  individuals  was  25.1  hours  for  math  and  27.6  hours  for  science  (the  medians  were  15.0  hours 
for  math  and  10.3  hours  for  science).  Combining  math  and  science,  the  sum  of  individual- 
level  participation  over  four  years  ranged  from  0.5  to  306  hours  but  was  in  the  range  of  five 
to  50  hours  for  most  participants.  During  years  3  and  4  of  the  project,  the  average  hours  of 

Figure  4.1 

Trends  in  the  Number  of  Participants  in  MSP  Activities 


RAND  MG857-4. 1 
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Figure  4.2 

Trends  in  Average  Time  Spent  by  Participants  on  MSP  Activities 


RAND  MG857-4.2 

individual-level  participation  decreased  faster  in  math  than  in  science.  This  may  be  a  natu¬ 
ral  result  of  the  staggered  implementation  of  various  TLA  and  OSA  activities  in  math  and 
science. 

Across  districts,  the  number  of  participants  and  their  hours  of  participation  varied  consider¬ 
ably.  For  example,  one  district  had  three  participants,  one  who  participated  for  four  hours  and 
two  who  participated  for  44  hours  each;  a  second  district  had  83  participants,  with  75  percent 
participating  for  more  than  10  hours;  a  third  district  had  155  participants,  with  84  per¬ 
cent  participating  for  more  than  10  hours;  and  a  fourth  district  had  13  participants,  all  of 
whom  participated  for  more  than  10  hours.  Across  all  districts,  the  proportion  of  teachers  who 
participated  ranged  from  4  to  81  percent. 

Over  the  course  of  four  years,  district-level  leaders’  total  participation  in  math-related 
activities  ranged  from  92  to  1,702  hours.  Leaders’  total  participation  in  science-related  activi¬ 
ties  ranged  from  70  to  1,068  hours.  For  most  districts,  leaders’  total  participation  in  math 
was  greater  than  in  science,  and  math  participation  varied  more  than  science  participation 
(see  Figure  4.3).  For  teachers,  the  average  adjusted  participation  in  math,  9.4  hours,  was 
slightly  higher  than  in  science,  8.7  hours  (see  Figure  4.4)7  There  is  a  relatively  strong  correla¬ 
tion  between  the  district-level  measures  of  leaders’  and  teachers’  participation:  The  correlation 
between  total  leader  participation  and  adjusted  teacher  participation  is  0.69  in  mathematics 
and  0.70  in  science  (n  =  53,  p  <  0.05). 


7  As  described  in  the  section  “Development  of  Participation  Measures  for  Analysis,”  district-level  teacher  participation  was 
adjusted  to  account  for  large  variations  in  district  size. 
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Figure  4.3 

Participation  in  Math  and  Science  MSP  Activities  by  District  Leaders 


RAND  MG857-4.3 
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Figure  4.4 

Adjusted  Participation  in  Math  and  Science  MSP  Activities  by  Teachers 


RAND  MG857-4.4 
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Trends  in  Mathematics  and  Science  Achievement 

As  a  first  look  at  achievement  change  during  the  project,  the  evaluation  team  examined  overall 
proficiency  levels  in  MSP  school  districts.  This  revealed  an  increasing  trend  from  2003-04  to 
2006-07  in  the  percentage  of  students  performing  at  or  above  the  level  of  proficient  on  the 
PSSA  mathematics  test  (see  Table  4.2.).8  This  result  is  encouraging,  but  it  does  not  rule  out 
the  possibility  that  factors  other  than  participation  in  the  MSP  contributed  to  the  increase  in 
proficiency.  To  conclude  that  the  MSP  was  the  source  of  the  trend  would  require  knowing  how 
attainment  of  proficiency  would  have  evolved  had  the  project  not  been  undertaken.  Of  course, 
we  cannot  observe  such  data,  but  as  a  coarse  proxy  for  this  alternative  outcome,  we  examined 
the  trend  in  proficiency  levels  in  non-MSP  school  districts  in  the  state.  If  the  MSP  were  respon¬ 
sible  for  the  observed  trend  in  participating  districts’  student  proficiency,  the  trend  should  be 


Table  4.2 

PSSA  Mathematics  Proficiency  Levels  in  MSP  and  Non-MSP  Districts 


Year 

Grade 

MSP  Districts 

Non-MSP  Districts  in  Pennsylvania 

Percentage 

Std.  Dev. 

Min  (%) 

Max  (%) 

Percentage 

Std.  Dev. 

Min  (%) 

Max  (%) 

2003-04 

Overall 

55 

0.17 

13 

85 

60 

0.11 

15 

88 

Elementary 

64 

0.18 

17 

93 

66 

11.63 

19 

93 

Middle 

54 

0.20 

14 

85 

61 

13.01 

13 

92 

High 

48 

0.19 

3 

81 

52 

13.76 

6 

90 

2004-05 

Overall 

65 

0.15 

30 

90 

69 

0.09 

23 

91 

Elementary 

76 

0.13 

45 

96 

79 

0.09 

29 

97 

Middle 

59 

0.20 

14 

90 

66 

0.12 

21 

92 

High 

48 

0.19 

13 

81 

53 

0.13 

5 

89 

2005-06 

Overall 

67 

0.15 

36 

89 

71 

0.09 

23 

92 

Elementary 

77 

0.12 

49 

94 

79 

0.09 

29 

97 

Middle 

64 

0.17 

27 

93 

69 

0.11 

21 

93 

High 

51 

0.18 

8 

84 

53 

0.13 

7 

90 

2006-07 

Overall 

68 

0.15 

33 

92 

72 

0.09 

20 

92 

Elementary 

77 

0.13 

39 

95 

80 

0.09 

28 

96 

Middle 

66 

0.17 

27 

94 

72 

0.11 

15 

93 

High 

52 

0.18 

5 

82 

55 

0.13 

3 

90 

NOTE:  For  the  grade-level  breakdowns,  all  PSSA  exams  administered  in  grades  3-5  were  classified  as  elementary, 
those  administered  in  grades  6-8  were  classified  as  middle,  and  the  grade  11  exam  was  classified  as  high.  PDE 
began  administering  the  PSSA  in  additional  grade  levels  during  the  course  of  the  project,  so  the  number  and 
types  of  exams  included  in  the  table  vary  across  years,  as  discussed  in  the  text. 


This  analysis  includes  48  MSP  districts,  excluding  MSP  expansion  districts  that  joined  the  project  in  year  4. 
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absent  or  weaker  in  non-MSP  districts.9  The  results  revealed  that  the  proficiency  trend  from 
2003-04  to  2006-07  in  non-MSP  school  districts  was  similar  to  that  of  MSP  school  districts 
(see  Figure  4.5). 

Because  there  is  no  proficiency  standard  for  the  PROM/SE  science  achievement  test,  we 
calculated  the  average  of  district-level  standardized  science  scores.  Similar  to  the  trend  in  math 
scores,  the  results  showed  that,  over  the  past  four  years,  there  has  been  a  slight  improvement 
in  the  mean  of  these  district-level  average  scores  on  the  PROM/SE  assessment  (see  Table  4.3).10 

Figure  4.5 

Proficiency  Levels  in  MSP  and  Non-MSP  Districts 


□  MSP  districts  I  I  Non-MSP  districts 


NOTE:  Each  panel  shows  the  percentage  of  students  at  or  above  proficient  on  the  PSSA  assessments  in 
the  48  MSP  districts  and  other  districts  in  Pennsylvania  during  the  2003-04  through  2006-07  academic 
years.  For  the  grade-level  breakdowns,  all  PSSA  exams  administered  in  grades  3-5  were  classified  as 
elementary,  those  administered  in  grades  6-8  were  classified  as  middle,  and  the  grade  1 1  exam  was 
classified  as  high.  MSP  expansion  districts  that  joined  the  project  in  year  4  are  excluded  from  both  groups. 
The  figure  includes  all  PSSA  exams  administered  each  year.  PDE  began  administering  the  PSSA  in 
additional  grade  levels  during  the  course  of  this  project,  so  the  number  and  types  of  exams  included  in 
the  figure  vary  across  years,  as  discussed  in  the  text. 

RAND  MG8S7  4.5 


9  This  evidence  would  be  somewhat  weak,  however,  because  the  analysis  does  not  attempt  to  establish  that  the  MSP  and 
non-MSP  groups  were  similar  at  the  start  of  the  project  in  terms  of  important  factors  related  to  achievement  (such  as  back¬ 
ground  characteristics  of  the  students).  The  statewide  analyses  presented  later  in  this  chapter  attempt  to  better  control  for 
such  factors. 

10  Pennsylvania  districts  outside  of  the  MSP  project  did  not  administer  the  PROM/SE  assessment,  so  no  statewide  science 
achievement  comparison  was  feasible. 
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Table  4.3 

MSP  District  Science  Assessment  Scores 


Year 

Mean  District-Level 
Average  Score3 

Std.  Dev.  of 
District-Level 
Average  Scores 

Number  of 
Districts 

Minimum 
District-Level 
Average  Score 

Maximum 
District-Level 
Average  Score 

2003-04 

-0.06 

0.32 

47 

-1.03 

0.56 

2004-05 

-0.05 

0.30 

46 

-0.76 

0.46 

2005-06 

0.04 

0.31 

47 

-0.64 

0.56 

2006-07 

0.06 

0.32 

45 

-0.82 

0.62 

a  Student-level  scores  for  each  form  of  the  PROM/SE  assessment  were  standardized  so  that  the  2003-04  scores 
had  a  mean  of  zero  and  standard  deviation  of  one.  The  district  average  is  the  mean  of  the  standardized  student 
scores,  and  the  values  reported  here  are  the  means  of  the  district  averages. 


Because  non-MSP  districts  do  not  administer  the  PROM/SE  assessment,  the  evaluation  team 
was  unable  to  compare  these  results  with  those  of  other  districts  in  the  state,  so  it  is  not  known 
whether  the  increases  observed  in  MSP  districts  reflect  statewide  trends  as  they  do  in  math. 


Relationship  of  MSP  Participation  to  Student  Achievement 

Achievement  models,  discussed  earlier  and  in  Appendix  A,  tested  whether  participation,  as 
measured  by  the  individual-  and  district-level  participation  measures,  is  related  to  student 
achievement  after  controlling  for  pre-MSP  test  scores  and  other  predictors  of  future  achieve¬ 
ment.  These  analyses  focused  on  students  for  whom  the  database  contained  pre-MSP  and  post- 
MSP  test  scores.  Appendix  A  contains  details  about  the  models  we  ran,  the  variables  included 
in  the  models,  and  full  results  from  each  model. 

Relationship  of  Participation  to  Math  Achievement 

Table  4.4  summarizes  the  cohorts  on  which  each  type  of  model  was  examined.  Students 
in  cohorts  G  and  J  had  math  achievement  scores  in  2003-04  (the  last  pre-MSP  year)  and 
2006-07  (three  years  after  MSP  implementation).  The  time  span  between  the  two  tests  gave 
these  students  the  maximum  amount  of  exposure  to  educators  who  had  undertaken  MSP 
training.  In  the  student-  and  district-level  analyses,  additional  cohorts  were  examined  where 
the  pre-  and  post-tests  spanned  fewer  years  and  for  which  the  pretest  was  not  a  true  pretest 
because  it  was  obtained  after  project  implementation  had  begun.  Analyses  of  cohorts  K  and 
M  cover  two  years  of  implementation,  and  those  of  cohorts  L  and  N  cover  one  year.  In  the 
student-level  analyses,  the  lack  of  a  sufficient  number  of  students  linked  to  educators  prohibited 
analyses  of  cohorts  J  and  M. 

In  total,  four  cohorts  were  examined  in  student-level  achievement  models,  six  cohorts 
were  examined  in  the  district-level  achievement  models,  and  two  cohorts  were  examined  in 
the  statewide  comparison  models.  As  shown  in  Table  4.4,  for  two  cohorts,  one  of  the  analy¬ 
ses  detected  a  significant  positive  relationship  between  participation  of  certain  educators  and 
achievement.  However,  in  both  cases,  the  same  models  produced  less  positive  results  for  other 
educators,  as  discussed  in  the  table  notes.  Moreover,  the  positive  findings  were  not  replicated  in 
additional  cohorts  or  types  of  analyses  (student-level,  district-level,  or  statewide  comparison). 
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Table  4.4 

Summary  of  Mathematics  Achievement  Models 


Cohort 


Mathematics  Analyses 

G 

J 

K 

L 

M 

N 

Student-level  models 

+a 

District-level  models 

Statewide  comparison  models 

+b 

NOTE:  Shaded  cells  indicate  cohorts  for  which  models  were  run.  Cells  with  plus-signs  indicate  models  in  which  a 
significant  positive  effect  was  detected  (p  <  0.05,  not  adjusted  for  multiple-hypothesis  testing).  Refer  to  Table  4.1 
for  additional  information  about  the  cohorts  and  Appendix  A  for  technical  details  about  the  models. 


a  This  model  linked  students  to  their  mathematics  teachers  in  sixth  and  seventh  grades  and  examined 
mathematics  achievement  over  this  two-year  period.  The  model  detected  a  positive  relationship  between  MSP 
participation  of  the  sixth-grade  teachers  and  student  achievement.  No  effect  was  detected  for  the  seventh- 
grade  teachers. 

b  This  model  examined  district-level  aggregate  math  achievement  over  a  three-year  period  when  students 
were  in  sixth  through  eighth  grades,  with  participation  of  leaders,  mathematics  teachers,  and  science  teachers 
included  as  covariates.  The  model  detected  a  positive  effect  for  leaders.  No  effect  was  detected  for  mathematics 
teachers.  These  results  were  stable  across  the  three  matched  comparison  groups  described  in  Appendix  A.  For 
science  teachers,  a  significant  negative  relationship  was  detected  for  two  of  the  three  comparison  groups. 


Overall,  we  conclude  that  (1)  these  analyses  do  not  find  a  coherent  relationship  between  educa¬ 
tor  participation  in  the  MSP  and  students’  math  achievement  and  (2)  the  few  significant  rela¬ 
tionships  that  were  detected  should  not  be  overinterpreted  or  overgeneralized. 

Relationship  of  Participation  to  Science  Achievement 

For  MSP  participation  and  science  achievement,  the  available  data  enabled  the  evaluation 
team  to  conduct  only  the  district-level  analysis.  The  student-level  and  statewide  comparison 
analyses  were  not  feasible  because  individual-level  science  achievement  scores  were  not  com¬ 
parable  and  the  comparison  districts  did  not  administer  the  same  science  achievement  test  as 
the  MSP  districts  did.  Based  on  the  available  data,  the  evaluation  team  examined  eight  cohorts 
of  students,  shown  in  Table  4.5.  Results  of  these  analyses  showed  no  significant  positive  rela¬ 
tionships  between  district-level  leaders’  or  teachers’  MSP  participation  and  students’  science 
achievement. 

Table  4.5 

Summary  of  Science  Achievement  Models 


Cohort 


Science  Analyses 

F 

G 

H  1 

J 

K 

M 

N 

District-level  models 

NOTE:  Shaded  cells  indicate  cohorts  for  which  models  were  run.  These  models  detected  no  significant  effects. 
(See  Table  4.1  for  additional  information  about  the  cohorts.) 
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Summary  of  Findings  Related  to  Student  Achievement 

In  summary,  this  chapter  examined  the  relationship  between  educators’  MSP  participation 
and  students’  math  and  science  achievement  at  the  individual  and  district  levels.  To  do  this,  the 
evaluation  team  developed  participation  measures  that  accounted  for  differences  in  the  poten¬ 
tial  impact  of  participants  who  play  a  leadership  role,  as  opposed  to  those  who  teach  but  do 
not  play  a  leadership  role.  The  evaluation  team  used  these  measures  in  three  distinct  analytic 
strategies  to  statistically  model  the  relationship  between  participation  and  achievement  and  ran 
these  models  on  multiple  cohorts  of  students.  Ideally,  the  results  of  these  varied  analyses  would 
align  to  form  a  coherent  picture  of  the  effect  of  MSP  participation  on  achievement  and  help  to 
overcome  the  limitations  of  this  project’s  nonexperimental  design. 

Results  of  these  analyses  showed  only  a  few  significant  relationships  between  MSP  partic¬ 
ipation  by  educators  and  student  math  achievement.  These  significant  relationships  appeared 
among  many  nonsignificant  findings  and  were  inconsistent  across  cohorts  and  analyses.  As  a 
result,  they  do  not  enable  us  to  draw  any  overall  conclusions  about  the  effects  of  the  MSP  on 
changes  in  math  or  science  achievement. 

There  are  some  possible  explanations  for  why  an  impact  of  educators’  MSP  participa¬ 
tion  on  learning  may  not  be  detectable  at  a  statistically  significant  level  in  these  analyses.  For 
example,  the  participation  measure  based  on  hours  is  imperfect  and  does  not  consider  such 
potentially  important  factors  as  the  engagement  of  educators  in  the  activities.  Moreover,  the 
PSSA  mathematics  and  PROM/SE  science  assessments  used  in  this  analysis  may  not  be  sensi¬ 
tive  enough  to  the  changes  in  student  learning  that  might  result  from  the  MSP  intervention. 
Finally,  teachers  may  not  have  had  sufficient  time  to  enact  in  their  classroom  teaching  what 
they  learned  in  MSP  professional  development  activities,  or  students  may  need  to  be  exposed 
to  these  changes  in  teaching  for  an  extended  period  before  showing  gains  on  assessments.  Part¬ 
ners  on  the  assessment  and  evaluation  team  continue  to  collect  and  analyze  achievement  data 
as  well  as  case  study  data  that  may  yield  useful  information  for  interpreting  the  achievement 
results. 


CHAPTER  FIVE 


Quality  of  the  Educator  Workforce 


The  second  MSP  project  goal  is  to  increase  the  quality  of  the  K-16  educator  workforce.  As 
discussed  in  Chapter  Three,  the  theory  of  action  stipulates  that  providing  teachers  with  oppor¬ 
tunities  to  deepen  their  content  and  pedagogical  knowledge  in  the  context  of  high-quality 
instructional  materials  will  result  in  better-prepared  teachers.  The  MSP  implemented  several 
activities  for  both  K-12  and  IHE  educators  to  achieve  this  goal.  Through  participation  in 
the  principal  seminars,  principals  were  exposed  to  reform-oriented  teaching  practices  and 
appropriate  methods  to  observe  and  frame  discussions  with  teachers.  K-12  teacher  leaders 
and  IHE  faculty  were  exposed  to  various  pedagogical  techniques  and  leadership  training  in  the 
TLAs.  K-12  educators  were  exposed  to  research-based  educational  materials  and  other  reform- 
oriented  practices  through  other  MSP  activities,  including  content  short  courses,  OSAs,  the 
teacher  fellow  program,  and  educator  networks. 

Because  it  is  difficult  to  measure  increases  in  the  quality  of  the  educator  workforce  directly, 
analysis  of  the  impact  of  the  MSP  focuses  on  short-  and  midterm  outcomes  associated  with  the 
quality  of  the  educator  workforce.  These  include 

•  increased  administrative  leadership  skills 

•  changes  in  district-  and  school-level  policies  and  practices 

•  increased  awareness  and  knowledge  of  research-based  instructional  practices  and 
materials 

•  changes  in  instructional  practices 

•  creation  of  professional  learning  communities. 

This  chapter  uses  survey  data  to  examine  changes  in  these  outcomes  over  the  course  of 
the  project,  as  well  as  the  relationship  of  MSP  participation  to  the  observed  changes.  The  chap¬ 
ter  also  examines  IHE  interview  and  observation  data  related  to  the  quality  of  the  educator 
workforce. 


Survey  Development  and  Analysis 

During  the  project,  we  administered  surveys  to  math  and  science  teachers  and  the  principal  at 
each  MSP  school.  Surveys  were  administered  twice  during  the  project,  and  we  refer  to  these  as 
baseline  and  follow-up,  respectively.  The  baseline  teacher  survey  was  administered  in  the  spring 
of  2004  after  one  year  of  MSP  planning  but  before  most  teachers  had  become  engaged  in  the 
project.  Of  the  approximately  3,200  teachers  in  participating  school  districts,  we  selected  a 
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sample  of  1,881,  stratified  by  level  (elementary,  middle,  or  high  school)  and  topic  area  (math¬ 
ematics,  science,  or  both).1  We  administered  a  follow-up  survey  to  teachers  three  years  later 
in  the  spring  of  2007.  All  who  responded  to  the  baseline  survey  were  included  in  the  follow¬ 
up  sample  if  they  were  still  teaching  in  the  district,  and  remaining  surveys  were  allocated 
using  a  stratified  sampling  method  similar  to  that  used  for  the  baseline  survey.  Responses  were 
received  from  1,241  (66  percent)  of  the  1,881  teachers  sampled  for  the  baseline  survey  and  1,574 
(79  percent)  of  the  1,988  teachers  sampled  for  the  follow-up  survey. 

The  baseline  principal  survey  was  administered  to  the  principal  of  each  MSP  school  in  the 
fall  of  2004,  and  the  follow-up  was  administered  in  the  fall  of  2006.  There  was  greater  than 
40-percent  turnover  in  principals  over  this  two-year  period,  so  caution  is  warranted  in  inter¬ 
preting  changes  from  baseline  to  follow-up.  Observed  changes  may  accurately  reflect  changes 
in  the  views  and  attitudes  of  the  schools’  leadership  over  this  period  but  may  not  represent 
changes  in  the  views  or  attitudes  of  individuals.  Responses  were  received  from  142  (71  percent) 
of  the  201  principals  sampled  for  the  baseline  survey  and  181  (92  percent)  of  the  197  principals 
sampled  for  the  follow-up  survey. 

The  teacher  and  principal  surveys  were  designed  to  capture  educators’  views  and  atti¬ 
tudes  about  science  and  mathematics  instruction;  current  practices  and  policies  in  curriculum, 
instruction,  assessment,  and  professional  development;  district  and  IU  support  for  improving 
schools;  and  MSP  project  impact.  The  teacher  survey  was  primarily  a  subset  of  questions  from 
the  Surveys  of  Enacted  Curriculum  produced  by  the  Council  of  Chief  State  School  Officers. 
We  selected  the  Surveys  of  Enacted  Curriculum  based  on  their  ability  to  measure  the  types  of 
teacher  practice  and  course-content  reforms  targeted  by  the  MSP  intervention  (Porter,  2002). 
Items  on  the  principal  survey  were  adapted  from  survey  instruments  developed  by  Horizon 
Research,  Inc.,  the  Center  for  the  Study  of  Teaching  and  Policy,  and  the  Center  for  Research 
on  the  Context  of  Teaching  and  from  principal  rubrics  developed  by  Richard  Halverson  at  the 
Wisconsin  Center  for  Education  Research.  The  majority  of  the  questions  on  the  baseline  and 
follow-up  surveys  were  identical,  but  some  new  MSP-specific  items  were  added  to  the  follow¬ 
up  teacher  and  principal  surveys  to  try  to  understand  the  impacts  of  MSP  activities  and  factors 
that  influenced  participation. 

We  made  use  of  imputation  to  enable  inferences  about  educators  in  all  MSP  schools  in 
some  analyses.  Specifically,  for  each  of  the  four  surveys  (baseline  and  follow-up,  teacher  and 
principal),  we  imputed  responses  that  were  missing  on  surveys  that  we  received.  Additionally, 
we  imputed  response  values  for  the  entire  survey  for  nonrespondents  to  the  baseline  and  follow¬ 
up  principal  survey  and  for  nonrespondents  to  the  follow-up  teacher  survey.  Those  surveys  each 
had  a  response  rate  of  71  percent  or  higher.  A  limitation  of  imputation  is  that  it  relies  on  data 
from  respondents  with  similar  characteristics  to  the  nonresponders.  If  the  nonresponders  are 
dissimilar  to  responders  on  measured  or  unmeasured  characteristics,  the  imputed  values  can 
be  unreliable.  This  concern  is  mitigated  somewhat  because  response  rates  were  relatively  high 
on  these  surveys,  and  our  inspection  of  imputation  results  did  not  reveal  any  anomalies.  Addi¬ 
tionally,  fully  imputed  surveys  for  nonrespondents  were  not  used  in  analyses  of  teacher  change 
from  baseline  to  follow-up  because  those  analyses  focused  on  798  teachers  who  responded  to 
both  the  baseline  and  follow-up  surveys  (the  798  respondents  were  roughly  evenly  divided 


i 


Elementary-school  teachers  typically  do  not  specialize  by  topic  area  and  teach  both  mathematics  and  science. 
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between  math  and  science  teachers).  Appendix  B  contains  additional  information  about  survey 
development,  sampling,  imputation,  and  analysis. 

Principal  Survey  Scales 

For  the  principal  survey,  we  used  factor  analysis  to  develop  27  scales  representing  subsets  of 
highly  correlated  items.  Table  5.1  displays  the  scales  along  with  their  reliability,  which  gener¬ 
ally  exceeds  0.75.  Eight  sets  of  scales  were  derived  from  questions  asked  twice  on  each  survey: 
once  regarding  science  and  once  regarding  math.  Reliabilities  are  reported  separately  for  those 
scales.  The  remaining  scales  are  not  specific  to  math  or  science. 

Teacher  Survey  Scales 

Similar  methods  were  used  to  develop  scales  for  the  teacher  survey.  Twelve  scales  were  formed 
from  items  appearing  on  both  the  baseline  and  follow-up  surveys,  and  seven  from  items  appear¬ 
ing  only  on  the  follow-up.  These  scales  are  shown  in  Table  5.2.  The  follow-up  survey  also 
included  items  inquiring  about  each  MSP  activity,  resulting  in  a  pair  of  scales  for  each  activity. 
Generally,  the  teacher  survey  scales  had  reliabilities  of  0.70  or  greater. 

Approaches  to  Survey  Analysis 

Using  these  scales,  we  performed  three  general  types  of  analyses:  (1)  descriptive  analysis  of 
changes  from  baseline  to  follow-up,  (2)  correlational  analysis  of  the  relationship  between  par¬ 
ticipation  in  MSP  activities  and  changes  from  baseline  to  follow-up,  and  (3)  correlational 
analysis  of  the  relationship  between  participation  in  MSP  activities  and  responses  on  the  fol¬ 
low-up  survey  for  items  that  were  administered  only  at  follow-up.  For  the  teacher  survey,  the 
descriptive  analysis  of  change  focused  on  the  798  teachers  who  responded  in  both  years.  For 
both  the  teacher  and  principal  surveys,  change  is  reported  as  the  mean  difference  (MD)  between 
the  two  observations.  This  enables  the  results  to  be  interpreted  in  the  units  of  the  survey  scale 
(usually  a  5-point  scale).  For  example,  an  MD  of  0.5  indicates  that  the  mean  response  on  the 
scale  at  follow-up  was  0.5  points  higher  on  the  survey  scale  than  the  mean  response  at  baseline. 
For  the  correlational  analysis  of  new  questions  on  the  follow-up  surveys,  we  analyzed  popula¬ 
tion  estimates,  in  some  cases  restricting  the  sample  to  those  respondents  who  had  attended  a 
given  MSP  activity. 

For  each  analysis,  we  began  with  the  usual  threshold  of  p-values  less  than  0.05  as  evidence 
of  Endings  not  attributable  to  chance,  but  we  adjusted  this  threshold  downward  due  to  the  vast 
number  of  statistical  tests  performed.  We  conducted  48  tests  comparing  baseline  to  follow-up 
teacher  survey  scales  and  36  tests  comparing  baseline  to  follow-up  principal  survey  scales.  For 
the  correlational  analyses,  we  conducted  66  tests  on  the  teacher  survey  and  28  on  the  princi¬ 
pal  survey.  We  used  the  Bonferroni  correction  for  each  family  of  tests.  Significant  findings  are 
indicated  with  an  asterisk  (*). 

Survey  analyses  used  the  participation  variables,  district  leadership  participation,  and 
adjusted  district-level  teacher  participation,  described  in  Chapter  Four.  However,  for  the 
teacher  survey  analyses,  we  separated  district  leadership  participation  into  two  components: 
the  school  level,  including  participation  by  such  leaders  as  principals,  who  are  assigned  to  a  spe¬ 
cific  school,  and  the  district  level,  including  participation  by  such  leaders  as  superintendents, 
whose  work  covers  multiple  schools.  In  addition,  for  the  principal  survey  analyses,  we  created  a 
school-level  variable  that  includes  only  participation  in  principal  seminars  because  that  activity 
was  specifically  designed  for  principals  and  was  intended  to  affect  their  attitudes  and  behaviors. 
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Table  5.1 

Principal  Survey  Scales 


Reliability  (a) 

Scale 

Math 

Science 

Scales  defined  separately  for  mathematics  and  science  instruction 

Knowledge  and  comfort  supporting  teachers  in  implementing  the  district  math 
(science)  curriculum  and  state  standards  (3  items) 

0.82 

0.85 

Views  on  the  importance  of  instructional  methods  that  emphasize  in-depth 
math  (science)  conceptual  understanding  (5  items) 

0.77 

0.82 

Views  on  the  importance  of  using  reform-oriented  approaches  to  math  (science) 
classroom  instruction  and  assessment  (8  items) 

0.80 

0.75 

Influence  of  state  and  federal  policies  and  standards  on  math  (science) 
instruction  (5  items) 

0.80 

0.84 

Influence  of  district  and  school  policies  and  reward  structures  on  effective  math 
(science)  instruction  (7  items) 

0.84 

0.89 

Influence  and  availability  of  instructional  resources  and  importance  placed  on 
math  (science)  instruction  (8  items) 

0.84 

0.86 

Views  on  ways  to  support  math  (science)  teachers  (3  items) 

0.89 

0.91 

Comfort  in  serving  as  math  (science)  instructional  leader  as  opposed  to  an 
administrative  manager  (5  items) 

0.84 

0.90 

Scales  common  to  the  mathematics  and  science  surveys 

Frequency  of  observing  classrooms  and  interacting  with  teachers  and  students 
(7  items) 

0.84 

Importance  of  allocating  resources,  building  learning  communities,  and 
focusing  on  assessments  (11  items) 

0.94 

Efforts  to  monitor  the  impact  and  involvement  of  teachers  in  professional 
development  (5  items) 

0.86 

Influence  on  policies  that  allow  teachers  time  to  engage  in  professional 
development  (4  items) 

0.65 

Influence  on  policies  related  to  professional  days  (3  items) 

0.93 

District  support  for  school  improvement  (10  items) 

0.90 

District  support  for  high  academic  standards  (10  items) 

0.95 

Role  of  the  IU  and  MSP  staff  in  supporting  the  MSP  (2  items) 

0.72 

Impact  of  MSP  activities  (9  items) 

0.92 

Value  of  university  courses  or  research-focused  professional  development 
attended  by  the  principal  (4  items) 

0.64 

Value  of  professional  development  received  through  professional  associations, 
mentoring,  or  peer  observation  (4  items) 

0.58 
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Table  5.2 

Teacher  Survey  Scales 


Reliability  (a) 

Scale 

Math 

Science 

Scales  from  items  appearing  on  both  the  baseline  and  follow-up  surveys 

Traditional  learning  activities  in  math  or  science  (6  items) 

0.67 

0.67 

Reform-oriented  learning  activities  in  math  or  science  (6  items) 

0.75 

0.73 

Student  problem  solving  or  reasoning  skills  in  math  (12  items) 

0.92 

— 

Student  demonstration  of  math  procedural  skills  (4  items) 

0.83 

— 

Student  demonstration  of  scientific  habits  of  mind  (9  items) 

— 

0.91 

Student  communication  of  science  (6  items) 

— 

0.81 

Assessment  activities  in  math  (8  items)  or  science  (5  items) 

0.69 

0.75 

Influence  of  district,  state,  and  national  standards  and  assessments  on  teaching 
(6  items) 

0.78 

Influence  of  student-  and  parent-related  factors  (4  items) 

0.71 

Comfort  and  preparedness  to  teach  at  the  appropriate  level  using  a  variety  of 
strategies  (6  items) 

0.83 

0.87 

Type  of  professional  development  completed  (7  items) 

0.71 

Relevancy  of  professional  development  to  teacher  and  school  needs  (5  items) 

0.89 

Focus  of  professional  development  on  instructional  approaches  and  individual 
student  learning  needs  (7  items) 

0.88 

Focus  of  professional  development  on  the  district  or  state  curriculum,  standards, 
and  assessment  data  (4  items) 

0.84 

Scales  from  follow-up  survey  items 

Vision  and  leadership  of  principal  (3  items) 

0.84 

0.84 

Principal's  support  for  professional  development  (3  items) 

0.81 

0.83 

Principal's  role  as  an  instructional  leader  in  the  school  (7  items) 

0.93 

0.94 

Teacher  views  of  the  atmosphere  in  the  school  and  support  for  collaboration 
and  reform 
(5  items) 

0.80 

0.83 

Informal  assessment  of  student's  knowledge  (5  items) 

0.78 

0.83 

Importance  of  instructional  methods  that  emphasize  in-depth  understanding 
and  inquiry-based  learning  (6  items) 

0.83 

0.85 

Importance  of  instructional  methods  that  emphasize  in-depth  understanding 
and  inquiry-based  learning  in  groups  (4  items) 

0.83 

0.83 

Scales  from  follow-up  survey  items  about  the  impact  of  MSP  activities  (repeated  for  each  activity) 

Impact  of  the  activity  on  the  awareness  and  understanding  of  how  students  think 
about  math  or  science  and  of  research-based  instructional  practices  (2-3  items) 

0.73-0.96 

Impact  of  the  MSP  activity  on  teaching  behavior  and  practices  (2-4  items) 
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The  choice  of  conducting  these  analyses  at  the  school  level,  rather  than  at  the  district  level,  was 
driven  by  the  observation  that  there  was  significant  variation  across  schools  within  districts, 
both  in  demographics  and  in  MSP  implementation.  Survey  sampling  was  designed  to  enable 
analysis  at  the  school  level. 


Findings  Reported  by  K-12  Educators 

The  remainder  of  this  chapter  discusses  teacher  and  principal  survey  results  and  information 
from  IHE  qualitative  data  examining  short-  and  midterm  outcomes  related  to  changes  in  the 
quality  of  the  educator  workforce.  We  begin  with  a  discussion  of  principal  and  teacher  impres¬ 
sions  of  the  overall  impact  of  the  MSP.  This  is  followed  by  an  examination  of  principal  leader¬ 
ship  and  support  for  reform-oriented  practices,  as  well  as  changes  in  school-  and  district-level 
policies.  Next,  changes  reported  by  teachers  on  their  instructional  influences  and  activities 
are  documented,  including  any  changes  reported  in  the  type  or  quality  of  professional  devel¬ 
opment  they  received  or  the  creation  of  professional  learning  communities.  Finally,  the  IHE 
qualitative  data  are  used  to  examine  the  impact  of  the  MSP  on  the  quality  of  the  educator 
workforce  at  IHEs. 

Overall  Impact  of  the  MSP 

Baseline  and  follow-up  principal  surveys  asked  about  the  influence  of  the  MSP  overall  on 
mathematics  and  science  instructional  practices,  as  well  as  the  impact  of  individual  MSP  activ¬ 
ities  on  instructional  practices.  The  questions  about  individual  MSP  activities  formed  one 
scale.  Overall,  principal  opinions  on  this  topic  did  not  change  significantly  from  baseline  to 
follow-up;  however,  principals  who  completed  the  principal  seminars  reported  greater  increases 
than  did  other  principals  from  baseline  to  follow-up  on  both  the  overall  impact  of  the  MSP 
and  a  scale  measuring  the  combined  impact  of  individual  MSP  activities  (MD  =  0.6*  for  both). 
In  addition,  on  the  follow-up  survey,  among  principals  who  reported  attending  the  principal 
seminars,  80  percent  reported  that  this  activity  had  influenced  their  views  or  behaviors  as 
principals. 

In  the  follow-up  teacher  survey,  teachers  who  participated  in  MSP  activities  reported 
their  perceptions  of  the  impact  of  these  activities.  Overall,  teachers  agreed  that  MSP  activities 
caused  them  to  increase  their  awareness  and  understanding  of  mathematics  or  science  concepts 
and  how  students  think  about  math.  They  also  agreed,  though  not  as  strongly,  that  the  MSP 
activities  helped  them  to  change  their  teaching  practices  and  motivated  them  to  seek  further 
information  or  training.  These  results  are  reported  by  MSP  activity  in  Table  5.3. 

Principal  Leadership 

Both  the  principal  and  teacher  surveys  contain  measures  of  principal  leadership  characteris¬ 
tics,  including  attitudes  and  behaviors  of  principals  and  teacher  views  of  the  support  provided 
by  the  principal.  In  general,  both  the  teacher  and  principal  survey  results  suggest  that  princi¬ 
pals  are  exhibiting  leadership  in  terms  of  creating  supportive  environments  to  effect  change, 
though  responses  related  to  principals  as  instructional  leaders  were  not  as  strongly  positive. 
These  results  are  interesting  because,  early  in  the  implementation  of  the  MSP,  the  project’s 
leaders  observed  that  support  from  the  administrative  school  leaders  was  important  in  facilitat¬ 
ing  changes  in  K— 12  teaching.  Recognizing  that  the  principal  seminars  can  be  influential  in 
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Table  5.3 

Teacher  Responses  Regarding  the  Impact  of  MSP  Activities  in  Which  They  Participated 


Follow-Up  Survey  Only 


Increased  Awareness  and  Understanding  Helped  Me  Change  My 

of  Math/Science  Concepts  and  How  Teaching  Practices  and  Seek  Further 

Students  Think  About  Math/Science  Information/Training 


Activity  Math  Science  Math  Science 


Leadership  Action 
Academies 

1.86  (0.08) 

2.09  (0.08) 

2.17  (0.08) 

2.17  (0.09) 

Teacher  Leadership 
Academies 

1.75  (0.05) 

1.95  (0.07) 

2.16  (0.06) 

2.14  (0.06) 

On-site  academies 

1.92  (0.04) 

2.21  (0.04) 

2.29  (0.04) 

2.42  (0.04) 

Educator  networks 

1.86  (0.06) 

1.80  (0.06) 

2.29  (0.07) 

2.18  (0.07) 

Content  short  courses 

1.74  (0.11) 

1.81  (0.07) 

2.09  (0.10) 

1.88  (0.07) 

NOTE:  Responses  are  based  on  a  5-point  Likert  scale  (1  =  strongly  agree,  5  =  strongly  disagree).  Values  in  the  table 
are  means  (and  standard  errors)  estimated  for  the  population  of  all  teachers  in  MSP  districts. 


gaining  this  administrative  support,  the  project  began  requiring  principals  to  attend  these 
seminars  as  a  prerequisite  for  new  districts  joining  the  MSP. 

From  baseline  to  follow-up,  the  principal  surveys  measured  changes  in  principals’  views 
of  the  importance  of  a  strong  professional  learning  community  and  comfort  in  the  role  of 
instructional  leader  (see  Table  5.4).  (Note  that  higher  values  on  these  scales  are  more  posi¬ 
tive  responses,  the  opposite  of  those  in  Table  5.3.)  First,  on  whether  developing  a  professional 
learning  community  encourages  effective  science  instruction,  principal  responses  increased  on 
a  5-point  scale  ranging  from  “inhibits  effective  instruction”  to  “encourages  effective  instruc¬ 
tion”  (MD  =  0.4*).  On  a  scale  of  “not  comfortable”  to  “very  comfortable,”  principals  reported 
increases  in  comfort  serving  as  math  instructional  leaders,  including  discussing  concrete  exam¬ 
ples  of  instructional  practices  with  teachers,  observing  classrooms,  examining  student  work, 
and  providing  feedback  on  teaching  (MD  =  0.2*).  Significant  changes  were  not  observed  on 
any  of  the  other  scales. 

Questions  on  the  follow-up  teacher  survey  gathered  information  about  the  roles  of  princi¬ 
pals  as  school  leaders  and  spanned  three  survey  scales:  principal  leadership  and  vision  in  creat¬ 
ing  a  supportive  environment,  principal  support  for  and  engagement  in  professional  develop¬ 
ment,  and  the  principal’s  role  as  an  instructional  leader.  Results  from  these  scales  are  shown 
in  Table  5.5.  Teachers  generally  agreed  that  principals  create  a  supportive  school  environment. 
They  also  agreed  that  principals  are  supportive  of  teacher  professional  development.2  Teach¬ 
ers  did  not  agree  as  strongly,  however,  that  the  principal  takes  an  instructional  leadership  role, 
engaging  with  teachers  about  teaching  practices  through  observations  and  feedback.3 


2  This  scale  includes  whether  the  principal  actively  participates  in  teacher  professional  development,  provides  adequate 
time  for  professional  development,  and  provides  time  for  teachers  to  share  information  and  experiences. 

3  Items  in  this  scale  include  whether  the  principal  talks  to  the  teacher  about  his  or  her  instructional  practices,  observes  the 
teacher’s  classroom,  and  examines  student  work  for  evidence  of  learning. 
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Table  5.4 

Principal  Responses  Regarding  Leadership  Skills 


Baseline  Follow-Up 


Principal  Survey  Scales,  Principal  Leadership 

Math 

Science 

Math 

Science 

Views  on  ways  to  support  math  (science)  teachers 

(1  =  inhibits  effective  instruction,  5  =  encourages  effective  instruction) 

4.11 

(0.08) 

4.07 

(0.06) 

4.23 

(0.10) 

4.46* 

(0.07) 

Comfort  in  serving  as  math  (science)  instructional  leader  as  opposed 
to  an  administrative  manager 

4.52 

(0.03) 

4.55 

(0.03) 

4.71* 

(0.03) 

4.67 

(0.04) 

(1  =  not  comfortable,  5  =  very  comfortable) 

Frequency  of  activities  related  to  building  learning  communities,  3.11  (0.05)  2.93  (0.05) 

focusing  on  assessments,  and  effective  use  of  resources 
(1  =  never,  5  =  almost  daily) 


Frequency  of  observing  classrooms  and  interacting  with  teachers  3.68  (0.05)  3.58  (0.05) 

and  students  (1  =  never,  5  =  almost  daily) 


Frequency  of  principal  and  teacher  involvement  in  planning,  3.41  (0.06)  3.40  (0.05) 

presenting,  and  assessing  professional  development 
(1  =  never,  5  =  always) 


NOTE:  *  indicates  significant  difference  from  baseline  to  follow-up  after  correcting  for  multiple-hypothesis 
testing.  Values  in  table  are  means  (and  standard  errors)  estimated  for  the  population  of  all  principals  in  MSP 
districts. 


Table  5.5 

Teacher  Responses  Regarding  Principal  Leadership 


Teacher  Survey  Scales  on  Follow-Up  Survey,  Principal  Leadership  Math  Teachers  Science  Teachers 

Principal's  activities  create  a  supportive  environment  2.12  (0.03)  2.07  (0.03) 

Principal  provides  support  for  teacher  professional  development  2.23  (0.03)  2.18  (0.04) 

Principal  takes  an  instructional  leadership  role  2.72  (0.03)  2.69  (0.04) 

NOTE:  Responses  are  based  on  a  5-point  Likert  scale  (1  =  strongly  agree,  5  =  strongly  disagree).  Values  in  the  table 
are  means  (and  standard  errors)  estimated  for  the  population  of  all  teachers  in  MSP  districts. 


District  and  School  Policies  and  Practices 

Another  midterm  MSP  outcome  focuses  on  changes  in  district  and  school  policies  and  prac¬ 
tices.  The  principal  survey  inquired  about  the  potential  impact  of  the  MSP  on  policies  and 
practices  (see  Table  5.6).  Of  these  scales,  only  two  showed  significant  change  from  baseline 
to  follow-up.  Principal  responses  increased  on  a  scale  probing  whether  instructional  resources 
encourage  effective  instruction  in  math  and  science  (math  MD  =  0.2*;  science  MD  =  0.5*). 4 
Principal  responses  also  increased  on  a  scale  asking  whether  district  and  school  policies  encour¬ 
age  effective  science  instruction  (MD  =  0.5*);  for  mathematics  instruction,  there  was  not  a 
significant  change  on  this  scale. 

Awareness  and  Knowledge  of  Research-Based  Instructional  Practices  and  Materials 

The  follow-up  teacher  survey  asked  whether  a  variety  of  reform-oriented  practices  are  impor¬ 
tant  for  effective  teaching,  such  as  allowing  students  time  to  puzzle  through  problems,  inviting 


4  Examples  of  scale  items  include  the  importance  placed  on  mathematics  or  science,  access  to  computers  and  calculators, 
and  the  quality  of  district-adopted  instructional  materials. 
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Table  5.6 

Principal  Responses  Regarding  School  and  District  Policies  and  Practices 


Baseline 

Follow-Up 

Survey  Scales,  District  and  School  Policies  and  Practices 

Math 

Science 

Math 

Science 

Influence  and  availability  of  instructional  resources  and  importance 
placed  on  math  (science)  instruction  (1  =  inhibits  effective  instruction, 

5  =  encourages  effective  instruction) 

4.19 

(0.04) 

3.81 

(0.06) 

4.42* 

(0.04) 

4.29* 

(0.05) 

Influence  of  district  and  school  policies  and  reward  structures  on 
effective  math  (science)  instruction  (1  =  inhibits  effective  instruction, 

5  =  encourages  effective  instruction) 

3.67 

(0.13) 

3.38 

(0.06) 

3.93 

(0.05) 

3.88* 

(0.06) 

Influence  on  policies  that  allow  teachers  time  to  engage  in 

professional  development 

(1  =  no  influence,  5  =  a  great  deal  of  influence) 

3.43  (0.08) 

3.40 

(0.06) 

Influence  on  policies  related  to  professional  days 
(1  =  no  influence,  5  =  a  great  deal  of  influence) 

2.95  (0.09) 

2.89 

(0.08) 

District  support  for  school  improvement 
(1  =  strongly  disagree,  5  =  strongly  agree) 

3.95  (0.05) 

4.03 

(0.06) 

District  support  for  high  academic  standards 
(1  =  not  at  all,  5  =  a  great  deal) 

3.76  (0.05) 

3.87 

(0.05) 

Influence  of  state  and  federal  policies  and  standards  on  math 
(science)  instruction  (1  =  inhibits  effective  instruction, 

5  =  encourages  effective  instruction) 

3.81 

(0.07) 

3.48 

(0.16) 

3.96 

(0.06) 

3.84 

(0.07) 

NOTE:  *  indicates  significant  difference  from  baseline  to  follow-up  after  correcting  for  multiple-hypothesis 
testing.  Values  in  the  table  are  means  (and  standard  errors)  estimated  for  the  population  of  all  principals  in  MSP 
districts. 


students  to  express  their  thoughts  and  ask  questions,  and  observing  and  listening  to  students 
interact.  These  questions  formed  three  classroom  practices  scales  on  which  teachers  generally 
reported  that  the  activities  are  important  for  effective  teaching  (Table  5.7). 

Participating  in  the  MSP’s  on-site  professional  development  led  by  teacher  leaders  is  one 
of  the  primary  mechanisms  for  teachers  in  the  MSP  to  be  exposed  to  reform-oriented  practices. 
On  both  the  baseline  and  follow-up  surveys,  teachers  reported  on  aspects  of  the  professional 
development  in  which  they  had  participated  during  the  previous  two  years  (see  Table  5.8). 
Math  and  science  teachers  reported  a  significant  increase  in  the  frequency  that  the  profes¬ 
sional  development  was  relevant  to  their  needs  and  the  needs  of  their  school  (math  and  science 
MD  =  0.1*).  Items  in  this  scale  inquire  about  how  often  the  professional  development  activi¬ 
ties  in  which  the  teachers  participated  were  designed  to  support  school  improvement  plans, 
were  consistent  with  the  teachers’  own  goals  for  development  and  their  department’s  plan  to 
improve  teaching,  were  based  on  what  they  had  learned  in  previous  professional  development, 
and  were  followed  up  with  activities  that  built  on  what  they  had  learned.  In  addition,  at  follow¬ 
up,  math  and  science  teachers  reported  that  the  professional  development  had  greater  emphasis 
than  at  baseline  on  instructional  approaches  and  individual  student  learning  needs  (math  and 
science  MD  =  0.1*).5  The  changes  reported  by  teachers  on  both  these  scales  are  consistent  with 
MSP  activities.  For  instance,  the  district  LATs  are  encouraged  to  develop  a  district-wide  plan, 


^  This  scale  asks  how  much  emphasis  the  professional  development  activities  place  on  such  topics  as  instructional 
approaches;  in-depth  study  of  concepts  in  math  or  science;  study  of  how  children  learn  topics,  including  individual  differ¬ 
ences  in  learning  and  meeting  the  learning  needs  of  special  populations;  and  classroom  assessment. 
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Table  5.7 

Teacher  Responses  Regarding  Classroom  Practices 


Teacher  Survey  Scales  on  Follow-up  Survey,  Classroom  Practices 

Math  Teachers 

Science  Teachers 

Importance  of  informal  assessments  of  student's  knowledge 

1.39  (0.01) 

1.46  (0.02) 

Importance  of  instructional  methods  that  emphasize  in-depth 
understanding  and  inquiry-based  learning 

1.39  (0.01) 

1.40  (0.02) 

Importance  of  instructional  methods  that  emphasize  in-depth 
understanding  and  inquiry-based  learning  in  groups 

1.46  (0.02) 

1.39  (0.02) 

NOTE:  Responses  are  based  on  a  5-point  scale  (1  =  very  important,  5  =  not  at  all  important).  Values  in  the  table 
are  means  (and  standard  errors)  estimated  for  the  population  of  all  teachers  in  MSP  districts. 


Table  5.8 

Teacher  Responses  Regarding  Professional  Development 


Baseline  Follow-Up 


Teacher  Professional  Development  Survey  Scales 

Math 

Science 

Math 

Science 

Frequency  of  professional  development  completed 
(1  =  never,  6  =  almost  daily) 

2.26 

(0.05) 

2.12 

(0.04) 

2.16 

(0.05) 

2.00* 

(0.04) 

Relevancy  of  professional  development  to  teacher  and  school  needs 
(1  =  never,  4  =  often) 

2.87 

(0.05) 

2.72 

(0.05) 

3.00* 

(0.05) 

2.85* 

(0.05) 

Focus  of  professional  development  on  instructional  approaches  and 
individual  student  learning  needs  (1  =  none,  4  =  great) 

2.47 

(0.05) 

2.24 

(0.05) 

2.61* 

(0.05) 

2.37* 

(0.05) 

Focus  of  professional  development  on  the  district  or  state  curriculum, 
standards,  and  assessment  data  (1  =  none,  4  =  great) 

2.98 

(0.05) 

2.54 

(0.05) 

3.03 

(0.05) 

2.65 

(0.05) 

NOTE:  *  indicates  significant  difference  from  baseline  to  follow-up  after  correcting  for  multiple-hypothesis 
testing.  Values  in  the  table  are  means  (and  standard  errors)  estimated  for  the  798  teachers  who  responded  to 
both  the  baseline  and  follow-up  surveys. 


and  MSP  OSAs  are  designed  to  build  sequentially  on  the  content  from  previous  academies  and 
are  intended  to  focus  on  reform-oriented  instructional  approaches,  such  as  those  measured  by 
these  survey  scales. 

Creation  of  Professional  Learning  Communities 

As  mentioned  earlier,  teachers  generally  agreed  that  principals  create  a  supportive  school  envi¬ 
ronment.  The  follow-up  teacher  survey  also  inquired  about  teacher  views  of  the  atmosphere  in 
the  school  and  support  for  collaboration  and  reform.  Some  of  the  items  on  this  scale  are  as  fol¬ 
lows:  support  by  colleagues  to  try  new  ideas,  teachers  regularly  share  ideas  and  materials,  and 
teachers  trust  each  other.  The  mean  response  of  both  math  and  science  teachers  on  this  5-point 
scale  was  between  “agree”  (2)  and  “neither  agree  nor  disagree”  (3).  The  mean  response  for  math 
was  2.47  (standard  error  =  0.03),  and  for  science  2.54  (standard  error  =  0.03). 

Instructional  Practices  of  K-12  Teachers 

An  important  midterm  goal  of  the  MSP  on-site  professional  development  and  other  activi¬ 
ties  is  to  improve  the  teaching  practices  of  math  and  science  teachers.  Four  scales  from  the 
teacher  surveys  address  student  learning  activities.  Each  scale  includes  four  to  12  survey  items. 
Although  the  scales  vary  slightly  for  math  and  science,  in  both  cases,  three  scales  measure 
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active  learning  and  reform-oriented  activities  and  one  measures  more  traditional,  passive  learn¬ 
ing  activities.  At  follow-up,  math  teachers’  responses  were  significantly  different  from  baseline 
on  two  scales,  while  science  teachers’  responses  were  not  significantly  different  on  any  of  the 
scales  (see  Table  5.9).  The  changes  in  student  learning  activities  reported  by  math  teachers  are 
consistent  with  greater  use  of  inquiry-based  learning  practices,  as  encouraged  by  the  MSP. 

Specifically,  math  teachers  reported  increases  in  the  amount  of  time  that  students  spend 
demonstrating  procedural  skills,  either  individually  or  in  pairs  or  small  groups  (MD  =  0.1*); 
this  includes  such  activities  as  students  working  on  problems  that  take  at  least  45  minutes  to 
solve  and  completing  proofs.  Math  teachers  also  reported  decreases  in  the  amount  of  time  that 
students  spend  on  more  traditional  types  of  learning  activities  (MD  =  0.1*),  such  as  watching 
the  teacher  demonstrate  how  to  solve  a  problem,  taking  notes  from  lectures  or  books,  complet¬ 
ing  worksheets,  working  individually  on  problems,  and  taking  tests.  However,  math  teachers 
did  not  report  significant  change  on  a  reform-oriented  learning  activity  scale,  which  includes 
having  the  students  demonstrate  solutions  to  the  class,  doing  a  math  activity  outside  of  the 
classroom,  working  in  pairs  or  small  groups,  and  using  manipulatives,  measurement  instru¬ 
ments,  or  data  collection  devices. 

Analysis  of  the  Relationship  of  MSP  Participation  to  Survey  Results 

Further  analyses  were  conducted  to  examine  the  relationship  of  educator  participation  in  the 
MSP  to  survey  responses.  Detecting  such  relationships  would  provide  supportive  evidence  that 
the  MSP  is  responsible  for  the  survey  response  changes  reported  earlier.  For  each  scale 
that  appeared  on  both  the  baseline  and  follow-up  surveys,  we  examined  the  correlations  between 
participation  and  changes  on  the  scale.  For  scales  appearing  only  on  the  follow-up  surveys, 
we  examined  correlations  between  participation  and  responses  at  follow-up.  For  the  principal 
survey,  in  addition  to  actual  participation  hours,  we  examined  the  correlation  between  whether 
the  principal  reported  attending  principal  seminars  and  changes  in  the  survey  scales. 


Table  5.9 

Teacher  Responses  Regarding  Student  Learning  Activities 


Baseline 

Follow-Up 

Teacher  Survey  Scales,  Student  Learning  Activities 

Math 

Science 

Math 

Science 

Traditional  learning  activities  in  math  or  science 

3.48 

(0.04) 

2.93 

(0.04) 

3.37* 

(0.04) 

2.90 

(0.04) 

Reform-oriented  learning  activities  in  math  or  science 

2.84 

(0.05) 

2.88 

(0.06) 

2.84 

(0.05) 

2.94 

(0.06) 

Student  problem  solving  or  reasoning  skills  in  math 

3.30 

(0.05) 

3.38  (0.05) 

Student  demonstration  of  math  procedural  skills 

1.88 

(0.06) 

2.02* 

(0.06) 

Student  demonstration  of  scientific  habits  of  mind 

3.67 

(0.06) 

3.67  (0.06) 

Student  communication  of  science 

2.72 

(0.06) 

2.80  (0.06) 

NOTE:  *  indicates  significant  difference  from  baseline  to  follow-up  after  correcting  for  multiple-hypothesis 
testing.  Responses  are  based  on  a  6-point  scale  (1  =  none,  6  =  almost  all  of  the  time).  Values  in  the  table  are 
means  (and  standard  errors)  estimated  for  the  798  teachers  who  responded  to  both  the  baseline  and  follow-up 
surveys. 
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In  sum,  94  tests  were  performed  and  only  two  revealed  a  significant  relationship.  As  dis¬ 
cussed  previously,  there  were  two  survey  measures  on  which  principals  responded  differently 
depending  on  whether  they  reported  attending  principal  seminars.  One  was  an  item  about 
the  overall  impact  of  the  MSP,  and  the  other  was  a  scale  about  the  impact  of  individual  MSP 
activities.  In  both  cases,  principals  who  reported  attending  the  seminars  also  reported  a  greater 
impact  on  instructional  practices  in  their  schools  (MD  =  0.6*  for  both).  We  examined  whether 
these  changes  in  survey  measures  also  showed  a  significant  correlation  with  hours  of  participa¬ 
tion  in  principal  seminars  (rather  than  principal  self-reports  of  attendance).  Those  models  did 
not  show  a  significant  relationship. 

In  summary,  these  analyses  were  not  successful  in  providing  supportive  evidence  that 
the  MSP  is  responsible  for  the  survey  response  changes  reported  by  educators.  However,  as 
discussed  in  Chapter  Four,  the  participation  measure  is  imperfect;  for  example,  it  does  not 
consider  the  engagement  of  participating  educators.  It  is  possible  that  participation  did  induce 
some  of  the  changes  observed  on  the  survey  scales  and  that  a  more  refined  participation  mea¬ 
sure  would  have  been  more  sensitive  in  detecting  this  relationship.  Future  analysis  of  data  from 
the  K-12  case  studies  may  also  help  to  illuminate  the  survey  findings. 


Contribution  of  IHEs  to  the  Quality  of  the  Educator  Workforce 

The  expected  contribution  of  the  IHEs  to  the  project’s  second  goal  of  increasing  the  quality  of 
the  educator  workforce  can  be  attained  through  multiple  routes.  The  primary  route  is  through 
IHE  participation  in  MSP  professional  development  activities.  As  a  result  of  participating  in 
expert  training  sessions  and  TLAs,  IHE  faculty  members  are  exposed  to  the  same  research- 
based  pedagogy  as  the  K-12  teachers.  This  experience  is  intended  to  lead  to  changes  in  IHE 
instructional  practices  as  STEM  and  education  faculty  members  incorporate  the  techniques 
into  their  classroom  practices.  Preservice  teachers  can  then  benefit  when  they  take  courses  from 
participating  STEM  and  education  faculty  members.  In-service  teachers,  who  participate  in 
TLAs  co-facilitated  by  IHE  faculty,  are  also  exposed  to  both  content  expertise  and  improved 
pedagogy.  A  secondary  route  for  IHEs  to  influence  the  quality  of  the  educator  workforce  is 
through  the  content  short  courses.  K-12  teachers  attend  in-depth  courses  on  mathematics  or 
science  topics  taught  by  IHE  faculty,  which  may  increase  their  knowledge  and  confidence  in 
teaching  these  topics.  Finally,  the  TF  program  is  another  route  that  is  expected  to  increase  the 
quality  of  the  educator  workforce  by  providing  K-12  teachers  with  immersion  in  the  higher 
education  environment.  TFs  spend  a  sabbatical  on  an  IHE  campus  and  participate  in  course- 
work  as  well  as  assist  a  faculty  member  in  the  revision  of  one  of  the  faculty  member’s  courses. 
Not  only  do  the  participants  benefit  from  their  experiences  as  fellows,  but  after  returning  to 
their  school  districts,  they  can  share  their  experiences  and  act  as  resources  for  other  teachers. 

The  following  sections  examine  some  of  the  contributions  of  IHEs  to  the  quality  of  the 
educator  workforce.  The  discussion  is  based  on  qualitative  analysis  of  IHE  data,  which  is 
described  in  detail  in  Appendix  C. 

Classroom  Instructional  Practices  at  IHEs 

Over  the  three-year  period  during  which  we  collected  data  from  IHEs,  we  interviewed  28  of 
the  approximately  55  STEM  faculty  members  who  were  actively  involved  in  the  MSP.  Some 
were  interviewed  multiple  times.  Overall,  the  majority  of  the  faculty  interviewed  reported 
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changes  in  instructional  practices.  Although  there  was  a  wide  range  of  descriptions  of  the  types 
of  changes  in  instructional  practices,  the  changes  appeared  to  focus  on  increasing  the  emphasis 
on  student- centered  learning,  which  included  group-oriented  classroom  activities  and  problem 
solving  in  class,  and  allowing  time  for  students  to  pursue  alternative  approaches.  In  addition, 
increasing  pedagogical  awareness  appeared  to  be  one  primary  factor  that  contributed  to  these 
changes  in  instructional  practices.  Each  of  these  areas  is  explored  in  greater  detail  next. 

Greater  Emphasis  on  Student-Centered  Learning.  A  number  of  IHE  faculty  reported 
altering  their  instruction  to  be  more  student- centered.  For  example,  one  IHE  mathematics  fac¬ 
ulty  member  said,  “[The  MSP]  helped  me  change  the  focus  from  teaching  to  student  involve¬ 
ment  ...  a  more  student-centered  approach  to  teaching.”  Similarly,  an  IHE  science  faculty 
member  said,  “In  thermodynamics,  I’m  much  more  likely  to  have  the  students  solving  prob¬ 
lems  in  teams  than  solving  them  on  the  board.  Four  years  ago,  I  would  never  have  them  solve 
the  problem.  I  let  them  make  mistakes  and  have  classmates  correct  them.”  There  are  indica¬ 
tions  suggesting  that  the  shift  toward  student-centered  instruction  has  had  positive  effects  on 
students.  IHE  faculty  members  reported,  “I  am  getting  better-quality  answers  back  from  stu¬ 
dents,”  “  [I]  get  students  to  think,”  and  “The  students  understand.” 

Classroom  observations  enabled  additional  insight  into  the  types  of  student- centered 
activities  used  by  faculty  members.  In  year  4,  we  observed  one  class  from  each  of  the  IHEs. 
Although  not  meant  to  be  representative,  these  four  classroom  observations  provide  practi¬ 
cal  examples  of  how  MSP-promoted  practices  were  used  in  IHE  classes.  In  many  cases,  IHE 
faculty  members  stated  that  these  practices  represented  a  departure  from  how  they  typically 
engage  their  students. 

Group  work  was  described  as  one  of  the  more  common  MSP-promoted  practices,  both 
in  interviews  with  faculty  members  and  in  classroom  observations.  Three  of  the  four  classes 
observed  used  small-group  discussion  during  class  time.  In  one  class,  small-group  discussion 
was  the  predominant  classroom  activity,  occupying  60  percent  of  class  time.  Small-group  dis¬ 
cussion  is  defined  as  activities  in  which  two  or  more  students  form  groups  and  engage  in  con¬ 
versation  with  each  other  about  specific  subject  matter.  Despite  the  common  use  of  this  instruc¬ 
tional  practice,  we  observed  that  its  purpose  varied  across  classes.  In  one  class,  small-group 
discussion  was  used  to  provide  an  opportunity  for  students  to  discuss  societal  implications  of 
scientific  knowledge,  whereas  in  another  class,  it  was  used  to  prepare  classroom  projects.  In  the 
classes  we  observed,  student  groups  were  self-selected.  When  asked  to  comment  on  the  effec¬ 
tiveness  of  student  grouping,  almost  all  faculty  members  reported  struggling  with  whether  to 
allow  students  to  self-select  into  groups  or  to  designate  the  grouping.  They  expressed  concern 
about  whether  all  students  benefited  from  the  strategy,  particularly  when  groups  bring  together 
students  who  are  not  equally  motivated. 

Increased  Pedagogical  Awareness.  Some  of  the  changes  in  instructional  practices  appear 
to  be  linked  to,  or  a  result  of,  increased  awareness  of  teaching  practices.  As  one  faculty  member 
explained,  “Being  involved  in  the  MSP  has  focused  faculty  on  elementary  and  secondary 
education.  Consequently,  attitudes,  understanding,  and  interest  in  educational  aspects  have 
increased.”  One  faculty  member  explained  that  her  participation  in  the  MSP  caused  her  to  be 
“thinking  about  teaching  all  the  time.”  Additionally,  some  faculty  members  reported  increased 
awareness  of  nationally  recognized  best  practices:  “I’m  more  aware  of  best  practices  at  the 
national  level.  I  realize  [that  we]  are  not  alone  in  facing  these  challenges,  and  we  can  be  part  of 
a  national  solution  to  these  issues,  which  is  rewarding.” 
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Faculty  members  also  reported  that  their  participation  in  the  MSP  increased  their  aware¬ 
ness  of  the  cognitive  development  of  their  students.  For  example,  one  faculty  member  said,  “I 
can  understand  my  students’  learning  better.  I  am  more  aware  of  the  education  theory  behind 
things.”  Similarly,  another  faculty  member  reported,  “Overall,  the  way  it’s  impacted  my  teach¬ 
ing  is  an  awareness  of  different  learning  styles.”  Increased  awareness  in  cognitive  develop¬ 
ment,  best  practices,  and  pedagogy  are  important,  as  these  may  be  associated  with  changes  in 
instructional  practices,  particularly  in  instances  in  which  this  awareness  may  have  been  limited 
prior  to  the  MSP. 

This  increase  in  pedagogical  awareness  varied  among  the  STEM  and  education  faculty. 
For  STEM  faculty,  we  noted  both  increased  awareness  of  education  theory  that  supports 
instructional  practices  and  increased  knowledge  of  innovative  practices  that  exist  for  teaching 
science  and  math.  Education  faculty  members  reported  benefits  from  increased  conversation 
about  pedagogy  with  STEM  colleagues  who  are  more  attuned  to  the  national  importance  of 
these  topics.  For  example,  one  education  faculty  member  said,  “The  biggest  benefit  for  me  is 
that  people  come  right  here;  we  can  be  the  hosts  of  a  lot  of  these  things,  and  we  get  the  ben¬ 
efit  of  materials  and  experts  in  the  held.  These  are  benefits  for  both  me  and  the  institution.” 
Another  reported, 

I  never  thought  I’d  be  interacting  with  content  faculty.  This  process  has  gone  through 
growing  curves.  I  am  now  in  constant  contact  with  the  math  department  and  with  science 
instructors.  I  meet  with  math  instructors  weekly.  Partnering  with  other  schools  [in  math 
and  science]  wouldn’t  have  happened  otherwise. 

Another  education  faculty  member  reported  that  increased  collaboration  and  communication 
with  faculty  in  the  mathematics  department  showed  her  that  “my  math  methods  are  somewhat 
effective,  [and]  now  I  can  see  how  I  can  improve  them.” 

Ultimately,  it  appeared  that  benefits  accrued  to  both  STEM  and  education  faculty  mem¬ 
bers  through  their  MSP-induced  collaborations:  STEM  faculty  members  gained  awareness  of 
instructional  practices,  and  education  faculty  members  received  greater  insight  into  the  per¬ 
spectives  of  content  experts. 

Practices  Related  to  Student-Teacher  Placements 

One  of  the  goals  of  the  MSP  is  to  place  student  teachers  from  the  IHEs  with  K-12  teachers  who 
have  participated  in  MSP  training.  Placing  student  teachers  with  MSP-participating  teachers  is 
intended  to  afford  them  opportunities  to  be  mentored  on  MSP-promoted  practices  and  to  be 
fully  supported  in  their  first  experiences  in  implementing  them.  Student  teaching  with  MSP- 
participating  teachers  is  expected  to  improve  the  quality  of  the  educator  workforce  by  increas¬ 
ing  the  pipeline  of  new  teachers  who  are  knowledgeable  about  and  comfortable  implementing 
the  research-based  practices  disseminated  by  the  MSP. 

Interviews  with  IHE  faculty  about  student-teacher  placement  indicated  that  this  is  one 
of  the  more  challenging  aspects  of  the  partnership  because  of  the  number  of  variables  that 
influence  it — many  of  which  are  outside  the  control  of  the  IHEs.  Interview  data,  particularly 
from  year  2,  indicate  that  the  locus  of  control  for  student-teacher  placement  resides  primarily 
with  the  school  districts.  As  explained  by  one  faculty  member,  “It’s  really  a  question  of  which 
schools  are  willing  to  take  them.  It  would  be  ideal  to  have  our  students  placed  in  [MSP]  schools 
that  we’re  working  with.”  The  willingness  on  the  part  of  schools  to  accept  student  teachers 
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from  a  particular  IHE  appears  to  be  governed  by  complex  rules  and  policies,  local  school  dis¬ 
trict  politics,  and  competition  from  IHEs  other  than  those  in  the  MSP.  Moreover,  the  signifi¬ 
cance  of  each  of  these  factors  varies  by  districts.  As  one  student-teacher  placement  coordinator 
explained,  “We’re  at  the  mercy  of  where  the  districts  can  put  the  student  teachers.  Sometimes 
they  have  their  own  rules;  for  example,  teachers  can’t  have  a  student  teacher  for  two  years  in  a 
row  in  some  districts.” 

Placing  student  teachers  in  a  particular  school  district  is  just  the  first  hurdle,  in  some 
instances.  Ensuring  that  they  are  placed  with  a  specific  teacher  can  be  even  more  difficult. 
Some  faculty  members  attributed  this  primarily  to  the  limited  authority  of  the  individual 
IHE: 


We  may  be  able  to  get  our  students  into  a  particular  school  district,  but  we  can’t  dictate 
which  teacher  they  are  assigned  to.  If  the  teachers  in  the  MSP  program  don’t  solicit  a  stu¬ 
dent  teacher,  we  can’t  assign  one  to  them.  Even  if  they  do  request  a  student  teacher,  we  don’t 
have  a  say  regarding  which  student  teacher  they  are  assigned. 

The  complexity  of  matching  student  teachers  with  specific  teachers  in  a  school  is  com¬ 
pounded  by  the  difficulty  of  accessing  information  on  which  teachers  have  participated  in 
the  MSP  in  any  given  school  district.  Many  student-teacher  coordinators  acknowledged  not 
knowing  which  teachers  participated  in  the  MSP.  In  addition  to  these  school-level  factors, 
the  preferences  of  the  student  teachers  themselves  also  influence  the  placements.  These  factors 
include  geographic  location,  as  student  teachers  often  prefer  to  teach  in  schools  close  to  where 
they  live;  salary  levels  in  the  schools,  since  they  are  often  offered  permanent  jobs  in  schools  in 
which  they  were  student  teachers;  and  whether  the  school  is  in  a  district  that  seems  likely  to  be 
hiring  the  following  year. 

Given  the  number  of  variables  that  influence  student  teacher  placement,  it  is  not  surpris¬ 
ing  that  many  faculty  members  emphasized  the  importance  of  developing  personal  relation¬ 
ships  with  the  districts  as  part  of  the  student-teacher  placement  process.  Faculty  members 
mentioned  spending  a  lot  of  time  developing  relationships  with  the  districts,  especially  with 
special  disciplines,  such  as  math  and  science,  in  which  it  may  be  more  difficult  to  place  student 
teachers.  However,  the  time  invested  in  developing  these  contacts  appears  to  be  wisely  spent,  as 
faculty  members  found  it  easier  to  place  students  when  there  were  good  personal  connections. 
Moreover,  having  personal  contacts  gave  the  IHEs  more  leverage,  as  explained  by  one  faculty 
member:  “I  had  a  lot  of  control  because  I  knew  the  teachers  and  had  a  network  of  people.” 

Changes  to  Student-Teacher  Placement  Practices.  As  of  year  4,  70  (53  percent)  of  the  ele¬ 
mentary  and  secondary  math  and  science  student  teachers  who  were  placed  in  partner  districts 
had  been  placed  with  MSP-participating  teachers.  Year  4  interview  data  suggests  that  there 
have  been  some  positive  changes  in  the  policies  and  practices  of  student-teacher  placement. 
Overwhelmingly,  data  from  each  of  the  four  institutions  suggest  that  there  are  now  explicit, 
intentional  efforts  to  place  student  teachers  with  MSP-participating  teachers.  As  one  faculty 
member  explained, 

There  is  a  more  extended  effort  this  year  to  place  student  teachers  with  teachers  who  have 
participated  in  MSP  activities.  Being  able  to  place  student  teachers  with  folks  who  have 
been  in  MSP  trainings  contributes  to  sustainability.  There  is  more  of  an  effort  to  complete 
the  cycle  and  to  have  student  teachers  see  the  methods  of  teaching  they’re  learning  rein¬ 
forced  in  the  school. 
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Many  other  faculty  members,  deans,  and  student-teacher  placement  coordinators  echoed  this 
statement,  suggesting  that  this  may  now  be  a  high  priority  in  student-teacher  placement:  “We’re 
much  more  aware  of  trying  to  place  student  teachers  in  MSP  districts  and  with  MSP  active 
teachers.  It’s  a  top  priority  to  us.” 

IHEs  used  various  strategies  to  identify  MSP-participating  teachers,  such  as  compiling 
a  list  of  teachers  who  had  participated  in  MSP  activities  in  the  past  or  who  were  currently 
participating  in  MSP  activities,  or  interviewing  teachers  before  placing  students  with  them  to 
determine  whether  they  had  participated  in  the  MSP  and  whether  they  utilized  inquiry-based 
practices  in  their  classrooms.  One  IHE  created  a  database  that  included  even  more  informa¬ 
tion  on  whether  teachers  had  (1)  participated  in  MSP  activities,  (2)  documented  use  of  inquiry- 
based  practices  in  their  classrooms,  (3)  expressed  willingness  to  host  a  student  teacher,  and 
(4)  received  positive  reviews  from  former  student  teachers.  In  addition  to  the  efforts  of  IHEs, 
the  MSP  PI  sent  a  letter  to  the  districts  emphasizing  the  importance  of  placing  student  teach¬ 
ers  with  teachers  from  the  MSP  leadership  cadre.  It  is  expected  that  these  concerted  efforts  will 
result  in  an  increase  in  strategic  student  placements. 

However,  some  faculty  members  felt  that  more  needed  to  be  done.  For  example,  one 
faculty  member  recommended  that  school  districts  develop  a  policy  to  ensure  that  student 
teachers  from  the  four  IHEs  will  be  placed  with  MSP-participating  teachers.  However,  it  may 
be  difficult  to  institute  this  policy  absent  evidence  that  these  placements  will  ultimately  lead 
to  changes  in  instructional  strategies  that  result  in  improved  learning  in  math  and  science. 
Although  many  faculty  members  speculated  on  the  advantages  of  these  placements,  it  was  also 
acknowledged  that  it  would  take  a  long  time  to  determine  whether  these  placements  will  make 
a  difference.  As  one  faculty  member  stated,  “It  takes  more  than  five  years  for  the  culture  to 
change  and  see  if  we’re  really  making  a  difference.  MSP  needs  to  be  a  long-term  progression, 
a  continuous  process.” 


Summary  of  Findings  Related  to  the  Quality  of  the  Educator  Workforce 

This  chapter  examined  data  from  teacher  and  principal  surveys  and  IHE  qualitative  data  for 
evidence  of  change  in  midterm  outcomes  related  to  the  quality  of  the  K-16  educator  work¬ 
force.  There  are  several  limitations  to  the  findings  reported  in  this  chapter.  First,  the  available 
data  do  not  directly  measure  workforce  quality,  but  instead  measure  midterm  outcomes  that 
may  be  associated  with  changes  in  workforce  quality.  Moreover,  the  bulk  of  evidence  comes 
from  self-reports  from  a  sample  of  participants  and  thus  is  subject  to  common  biases  associated 
with  such  data;  for  example,  if  respondents  answer  survey  or  interview  questions  inaccurately 
or  if  sampled  respondents  were  not  representative.  Finally,  the  measures  of  participation  used 
to  examine  links  between  survey  results  and  MSP  activities  are  imperfect,  capturing  time  par¬ 
ticipating  in  activities  but  not  the  quality  of  the  activities  or  the  engagement  of  educators. 

Generally,  educators  at  both  the  K— 12  and  IHE  levels  reported  changes  in  leadership  and 
instruction  that  are  consistent  with  the  MSP  theory  of  action.  K-12  teachers  reported  that  MSP 
activities  increased  their  awareness  and  understanding  of  math  and  science  concepts  and  how 
students  think  about  math  and  science,  and  helped  them  change  their  teaching  practices.  They 
also  reported  that  the  professional  development  in  which  they  participated  was  more  often  rel¬ 
evant  to  their  needs  and  was  focused  on  instructional  approaches  and  individual  student  learn¬ 
ing.  Principals  reported  that  the  principal  seminars  influenced  their  views  and  behaviors  as 


Quality  of  the  Educator  Workforce  51 


principals.  IHE  faculty  reported  a  greater  emphasis  on  student-centered  instruction  and  more 
awareness  of  different  pedagogical  techniques.  These  findings  suggest  that  the  activities  of  the 
MSP  may  be  having  the  intended  effect  on  midterm  outcomes  associated  with  increasing 
the  quality  of  the  educator  workforce.  However,  further  statistical  analyses  of  the  relationship 
between  participation  in  MSP  activities  and  survey  scales  tracking  key  midterm  outcomes  did 
not  provide  evidence  that  the  MSP  is  responsible  for  the  changes  reported  by  educators.  Data 
from  the  K— 12  case  studies  may  be  helpful  in  interpreting  these  findings. 


CHAPTER  SIX 


Sustainable  Partnerships 


The  third  goal  of  the  MSP  is  to  create  partnerships  that  are  sustainable  and  coordinated  by  the 
IUs.  These  partnerships  are  expected  to  build  intentional  feedback  loops  between  the  K— 12 
and  IHE  levels  that  tap  the  discipline-based  expertise  of  IHEs  and  improve  the  mathematics 
and  science  learning  experiences  of  all  K-12  and  undergraduate  students,  including  relevant 
preparation  for  preservice  teachers  (Shaneyfelt,  2006-2007).  The  MSP  plans  to  measure  the 
achievement  of  this  goal  by  examining  the  number  of  IHE  courses  revised  through  MSP  activ¬ 
ities,  the  sustained  involvement  of  IHE  faculty,  the  placement  of  student  teachers  in  MSP  dis¬ 
tricts,  and  the  financial  sustainability  of  having  the  IUs  cover  the  personnel  costs  of  the  MSP 
coordinators.  Because  the  MSP  project  continues,  the  extent  to  which  these  goals  have  been 
achieved  is  not  discussed  here.  Rather,  this  chapter  focuses  on  three  midterm  outcomes  asso¬ 
ciated  with  the  third  goal  of  creating  sustainable  partnerships — development  of  partnership, 
changes  in  institutional  policies  and  practices,  and  implementation  of  challenging  courses  via 
the  revision  of  IHE  courses. 


Partnership  Development 

Sustained  faculty  involvement  in  the  MSP  will  result,  in  large  part,  from  the  development  of 
partnerships  among  the  key  players.  Yet,  current  partnership  literature,  including  literature 
from  other  Math  and  Science  Partnerships  and  K-20  partnerships,  suggest  that  very  little  is 
understood  about  partnership  development  in  the  public  sector  and  the  characteristics  and 
factors  that  contribute  to  partnership  sustainability  (Boyell,  2007;  Clifford  and  Millar,  2008; 
Halliday,  Asthana,  and  Richardson,  2004;  Kingsley  and  Waschak,  2005;  Scherer,  2006;  E. 
Weiss,  Anderson,  and  Lasker,  2002).  Even  the  term  partnership  is  defined  ambiguously  or  not 
at  all  in  much  of  the  literature  (Kingsley  and  Waschak,  2005).  As  part  of  our  evaluation,  we 
examined  the  extent  to  which  partnership  development  occurred  within  the  MSP.  Our  work¬ 
ing  premise  conceptualized  partnership  and  the  process  of  partnership  development  as  evolv¬ 
ing,  rather  than  static.  Thus,  the  first  step  of  our  analysis  involved  operationalizing  the  terms 
partnership  and  partnership  development  in  order  to  establish  how  MSP  partners  view  partner¬ 
ship  and  determine  the  extent  of  partnership  development.  A  review  of  the  MSP  and  K— 20 
partnership  literature  provided  a  useful  taxonomy  that  conceptualizes  partnership  in  four  dis¬ 
tinct,  though  not  mutually  exclusive,  ways  (Kingsley  and  Waschak,  2005): 
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1.  entity-based,  in  which  partnerships  are  conceptualized  as  comprising  memberships, 
boundaries,  and  formal  and  informal  organizing  structures  designed  to  achieve  specific 
functions 

2.  agreement-based,  in  which  partnerships  are  conceptualized  as  a  set  of  predetermined 
goals  aimed  at  improving  performance  in  STEM  education 

3.  process-based,  in  which  partnerships  are  conceptualized  as  relationships  that  are  built  up 
over  time  to  enhance  levels  of  trust  and  cooperation 

4.  venue-based,  in  which  partnerships  are  conceptualized  as  simply  an  opportunity  to 
interact. 

We  used  this  taxonomy  of  partnership  conceptualizations  to  categorize  faculty  members’  defi¬ 
nitions  of  partnership. 

Similarly,  we  examined  the  partnership  literature  to  identify  relational  features  that  indi¬ 
cate  the  development  of  partnership.  A  literature  review  by  Clifford  and  Millar  (2008)  identi¬ 
fied  62  factors  associated  with  partnership  success.  Some  of  the  most  frequently  cited  factors 
included  open  communication,  joint  work  and  resource  exchange,  goal  focus,  and  trust  and 
respect  among  partners.  We  used  this  list  of  factors  to  identify  and  code  comments  from  IHE 
faculty  interviews  that  are  indicative  of  partnership  development. 

Some  key  findings  emerged  from  our  analysis:  (1)  the  majority  of  IHE  faculty  members 
have  process-based  conceptualizations  of  partnership;  (2)  partnerships  appear  to  have  devel¬ 
oped  between  IHE  and  K-12  educators,  between  IHEs,  and  between  departments  within 
IHEs;  and  (3)  there  appears  to  be  significant  variation  in  the  strength  of  the  partnerships.  Each 
of  these  findings  is  discussed  in  detail  next. 

Conceptualizations  of  Partnership 

Our  initial  analysis  sought  to  determine  how  well  Kingsley  and  Waschak’s  (2005)  MSP  part¬ 
nership  taxonomy  could  be  applied  to  IHE  faculty  members’  conceptualizations  of  partner¬ 
ship.  After  coding  faculty  member  responses  to  questions  about  the  meaning  of  partnership, 
we  found  that  a  majority  of  responses  fit  within  the  taxonomy.1  Approximately  83  percent  of 
faculty  interviewees  responded  with  process-based  conceptualizations,  in  which  partnerships 
are  conceptualized  as  relationships  built  up  over  time  to  enhance  levels  of  trust  and  coopera¬ 
tion.  For  example,  a  faculty  member  with  a  dual  appointment  in  education  and  science  said, 
“  [A  partnership  is  a]  network  and  resource  sharing;  letting  the  teachers  know  who  they  can  go 
to  and  rely  on  for  professional  development  opportunities.” 

Roughly  12  percent  of  IHE  faculty  members  reported  venue-based  conceptualizations  in 
which  partnership  is  considered  an  opportunity  for  interaction.  This  is  represented  in  the  fol¬ 
lowing  comment:  “It  means  now  you  know  some  people  in  high  schools;  if  they  have  a  problem 
they  might  call  you,  or  .  .  .  you  might  call  them.” 

Interestingly,  none  of  the  IHE  faculty  members  in  our  sample  conceptualized  partner¬ 
ship  as  agreement-based  or  as  entity-based.  This  differs  from  the  Kingsley  and  Waschak  study 
(2005),  which  found  that  nearly  one-third  of  their  respondents  described  conceptualizations 
of  partnership  that  were  agreement-based,  while  most  of  the  remaining  two-thirds  described 
conceptualizations  that  were  process-based.  These  differences  may  be  due  in  part  to  the  selec- 


1  These  partnership  conceptualizations  are  not  mutually  exclusive,  and  indeed,  several  IHE  faculty  members  had  responses  that 
appeared  to  fit  with  more  than  one  conceptualization  of  partnership.  In  those  cases,  we  coded  the  response  in  both  categories. 
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tion  criteria  we  used  in  analyzing  IHE  faculty  member  interviews.  Our  analysis  used  the  most 
recent  interview  for  each  person,  with  the  majority  of  these  conducted  in  year  3.  It  seems 
plausible  that  partnership  conceptualizations  develop  and  change  over  time,  starting  with  a 
conceptualization  of  partnership  as  a  formalized  agreement  and  eventually  maturing  into  a 
process-based  conceptualization.  Thus,  an  interesting  avenue  to  explore  as  the  project  draws  to 
a  close  is  whether  the  majority  of  IHE  faculty  members’  conceptualizations  become  process- 
based  by  the  end  of  the  project. 

Finally,  about  10  percent  of  faculty  members  either  did  not  answer  the  question  or  their 
responses  were  not  directly  related  to  the  meaning  of  partnership.  For  example,  a  response  of 
one  IHE  faculty  member  appeared  to  be  more  of  a  suggestion  about  how  the  MSP  could  be 
improved  rather  than  a  statement  on  the  meaning  of  partnership:  “Faculty  should  go  into  K-12 
classrooms  ...  so  that  they  can  see  what  K-12  education  looks  like.”  This  comment — though 
not  without  value — addresses  another  feature  of  partnership,  not  its  meaning.  Although  we 
searched  the  entire  length  of  these  IHE  faculty  interviews,  we  were  unable  to  determine  which 
conceptualizations  of  partnership  the  faculty  members  held. 

Partnership  Development  Between  K-12  and  IHEs 

Analysis  of  partnership  development  between  IHE  and  K-12  educators,  among  institutions, 
and  among  departments  within  institutions  was  based  on  faculty  members’  descriptions  of 
their  interactions  with  other  partners  and  whether  these  descriptions  included  factors  thought 
to  contribute  to  successful  partnerships  as  referenced  in  the  existing  literature  (e.g.,  open  com¬ 
munication,  joint  work  and  resource  exchange,  goal  focus,  trust  and  respect  among  partners). 
We  assumed  that  partnership  development  was  present  if  any  one  of  these  factors  were  described 
in  the  interview  data.  In  some  cases,  the  factors  were  mentioned  verbatim,  and  in  other  cases, 
similar  words  or  phrases  were  used  by  the  IHE  faculty  members;  for  example, 

•  open  communication:  talking,  contact,  email,  interactions,  sharing 

•  joint  work  and  resource  exchange:  learning,  collaboration,  MSP  work 

•  goal  focus:  common  goals,  shared  educational  ideals,  objectives 

•  trust  and  respect  among  partners:  admiration,  comfort,  one  of  us. 

The  interview  data  suggest  that  partnerships  developed  between  IHE  and  K-12  edu¬ 
cators,  between  IHEs,  and  between  departments  within  IHEs.  Figure  6.1  illustrates  several 
aspects  of  these  partnerships.  In  this  figure  and  those  in  the  following  sections,  we  use  the  fol¬ 
lowing  notations: 

1.  Bidirectional  arrows  represent  instances  in  which  there  were  reciprocal  descriptions 
of  partnership  between  IHE  and  K-12  educators  who  had  participated  in  the  TF 
program. 

2.  The  text  to  the  left  of  each  arrow  lists  the  partnership  indicators  used  by  both  IHE  and 
K-12  educators  to  characterize  the  relationship.  Except  where  noted,  members  of  both 
the  K-12  and  IHE  partners  mentioned  the  indicator. 

3.  The  thickness  of  the  arrow  means  that  at  least  three  K-12  educators  and  at  least  three 
educators  at  each  of  the  IHEs  used  the  partnership  indicators  in  characterizing  their 
relationship.  Because  the  number  of  interviews  varied  from  eight  to  14  per  IHE,  this 
represented  a  significant  percentage  in  some  cases. 
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Figure  6.1 

Partnership  Development  Between  IHE  Faculty  and  K-12  Educators  Who  Participated  in  the 
Teacher  Fellow  Program 
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All  the  IHE  and  K-12  educators  we  interviewed  referred  to  open  communication  and 
joint  work  in  describing  their  interactions  with  each  other.  Additionally,  trust  was  used  by 
educators  to  describe  the  relationships  between  K-12  institutions  and  two  of  the  IHEs,  and 
common  goals  was  used  by  educators  to  describe  the  relationship  between  K-12  institutions  and 
one  IHE.  We  also  hypothesize  several  indicators  of  the  strength  and  stability  of  the  relation¬ 
ships  between  partners.  One  of  these  is  whether  the  description  of  partnership  is  reciprocal. 
If  both  entities  describe  partnership  interaction  in  particular  terms,  it  is  likely  to  be  mutually 
reinforcing.  A  second  indicator  is  whether  descriptions  of  partnership  interactions  incorporate 
multiple  factors  (e.g.,  communication,  joint  work).  Finally,  a  third  indicator  is  whether  descrip¬ 
tors  of  partnership  development  were  reported  by  multiple  individuals  at  a  given  institution. 
All  of  these  hypothesized  indicators  of  partnership  strength  and  stability  were  observed.  Thus, 
we  conclude  that  the  partnership  development  between  IHE  and  K-12  educators  seems  stable 
and  strong. 

A  limitation  of  these  findings  is  the  select  group  of  K-12  educators  in  our  sample:  those 
who  had  participated  in  the  TF  program.  This  is  one  of  the  more  intensive  MSP  activities,  in 
which  IHE  faculty  and  K-12  teachers  work  together  for  at  least  one  semester,  and  in  some 
cases  as  long  as  a  year.  Moreover,  in  addition  to  their  participation  in  the  TF  program,  these 
K-12  teachers  also  interacted  with  IHE  faculty  in  other  MSP  activities,  such  as  the  TLAs. 
Thus,  K-12  educators  who  had  participated  in  the  TF  program  may  have  developed  partner¬ 
ships  with  IHE  faculty  that  are  not  representative  of  partnership  development  between  IHE 
faculty  and  non-TF  K-12  educators.  On  the  other  hand,  partnership  development  between 
IHE  faculty  and  non-TF  K-12  educators  could  be  equally  strong  because  other  MSP  activities 
also  attempt  to  foster  this  type  of  partnership.  By  virtue  of  our  restricted  sample,  we  cannot 
assess  the  partnership  that  may  have  developed  between  K-12  educators  and  IHE  faculty 
through  these  other  activities. 
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Partnership  Development  Across  IHEs 

We  also  found  evidence  of  partnership  development  between  faculty  members  of  different 
IHEs.  As  shown  in  Figure  6.2,  most  partnership  development  was  reciprocally  described 
(depicted  with  bidirectional,  solid  black  arrows).  However,  there  are  two  pairs  in  which  part¬ 
nership  development  was  described  by  only  one  of  the  partners  (IHE  B).  This  is  indicated  by 
unidirectional,  dashed  arrows.  The  number  of  IHE  faculty  members  who  described  partnership 
development  varied  across  the  pairs  of  institutions.  The  thickness  of  the  arrowed  lines  indicates 
the  number  of  faculty  members  describing  partnership  development.  The  thinnest  lines  indi¬ 
cate  partnership  development  described  by  one  IHE  faculty  member;  medium-thickness  lines 
indicate  partnership  development  described  by  two  faculty  members;  and  the  thickest 
lines  indicate  partnership  development  described  by  at  least  three  faculty  members  from  each 
IHE.  The  text  labels  in  Figure  6.2  show  the  factors  used  by  faculty  members  to  describe  the 
partnership  development.  The  most  frequently  described  partnership  development  factor  was 
open  communication.  Joint  work  was  also  used  to  describe  partnership  development  between 
most  pairs  of  institutions.  No  faculty  members  used  common  goals  or  trust  to  describe  partner¬ 
ship  development  among  IHEs. 

Partnership  Development  Across  Departments  Within  IHEs 

Partnership  also  appeared  to  develop  between  faculty  members  in  different  departments 
within  a  given  IHE.  Although  each  IHE  saw  slightly  different  department-level  partnership 
development,  there  are  several  common  patterns.  Figure  6.3  shows  a  representative  example  of 
department-level  partnership  development  from  one  IHE.  At  this  IHE,  partnership  develop¬ 
ment  was  described  between  faculty  members  in  the  education  and  math  departments,  the 
education  and  science  departments,  and  the  math  and  science  departments.  However,  the  only 

Figure  6.2 

Partnership  Development  Across  IHEs 
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Figure  6.3 

Partnership  Development  Between  Departments  at  One  IHE 
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one  of  these  that  was  described  reciprocally  was  between  the  education  and  math  depart¬ 
ments.  The  others  were  described  in  only  one  direction,  indicated  by  the  unidirectional,  dashed 
lines. 

Across  all  IHEs,  the  education  and  math  departments  were  the  only  department  pairs  in 
which  partnership  development  was  described  at  every  institution;  these  were  described  recip¬ 
rocally  in  each  case  (see  Table  6.1).  Faculty  members’  impressions  of  partnership  development 
between  math  and  science  departments  and  science  and  education  departments  were  either  not 
described  at  all  or  were  not  reciprocally  described. 


Table  6.1 

Patterns  of  Partnership  Development  Between  IHE  Departments 


Departments 

Math  and  Science 

Math  and  Education 

Science  and  Education 

IHE 

Factors 

Described 

Reciprocal 

Factors 

Described 

Reciprocal 

Factors 

Described 

Reciprocal 

A 

Communication 
Joint  work 

V 

Communication 
Joint  work 

V 

B 

Communication 
Joint  work 

V 

Communication 
Joint  work 
Trust3 

V 

C 

Communication 
Joint  work 

V 

Communication 
Joint  work3 

V 

D 

Communication 
Joint  work 

Communication 
Joint  work 
Trust 

V 

Communication 
Joint  work 
Goals 

Trust 

a  Factor  was  described  by  faculty  members  in  the  education  department  only. 
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Changes  in  Institutional  Policies  to  Support  Sustainability 

Sustained  MSP  participation  may  be  a  function  not  only  of  the  strength  of  relationships 
between  partners,  but  also  of  institutional  factors  that  can  support  or  inhibit  partnership.  For 
example,  the  literature  on  math  and  science  partnerships  indicates  that  institutions’  academic 
reward  systems,  leadership,  and  resources  play  significant  roles  in  sustained  participation  (Hora 
and  Millar,  2008;  Zhang  et  ah,  2008).  This  led  us  to  examine  the  IHE  interview  data  for  insti¬ 
tutional  factors  that  appeared  to  be  associated  with  partnership  development  between  institu¬ 
tions  in  the  MSP.  Table  6.2  shows  the  factors  that  emerged. 

This  list  of  factors  was  drawn  primarily  from  the  interview  data,  rather  than  being  a  tax¬ 
onomy  drawn  from  the  literature.  As  shown  in  the  table,  institutional  factors,  such  as  academic 
reward  systems,  campus  initiatives,  a  history  of  outreach  and  collaboration,  and  supportive 
leadership,  appeared  to  be  contributing  factors.  Conversely,  factors  that  appeared  to  inhibit 
partnership  included  unsupportive  academic  reward  systems  and  such  external  factors  as  indi¬ 
viduals’  health  problems,  faculty  or  administrative  turnover,  and  limited  resources,  such  as 
time  or  finances.  Among  these  factors,  the  academic  reward  system  appears  to  have  the  poten¬ 
tial  to  both  contribute  to  and  inhibit  partnership  development  and  was  one  of  the  factors  cited 
most  frequently  by  IHE  faculty  members.  The  following  section  explores  academic  reward  sys¬ 
tems  to  better  understand  how  they  vary  across  institutions  and  how  they  may  have  changed  to 
help  enable  the  sustainment  of  the  MSP  partnerships  beyond  the  term  of  the  project’s  grants. 

IHE  Academic  Reward  Systems 

Generally,  three  factors  are  considered  for  IHE  faculty  members  seeking  promotion  or  tenure: 
(1)  scholarship  and  research,  (2)  teaching  and  professional  development,  and  (3)  service  to  the 
institution  and  community.  However,  these  three  factors  are  not  weighted  equally  in  tenure 
and  promotion  decisions.  Many  IHEs,  particularly  research  universities,  consider  research  pro¬ 
ductivity  to  be  synonymous  with  success  (Boyer  Commission,  1998).  The  focus  on  research 
and  research  products  as  two  of  the  most  valued  outcomes  of  the  university  system  can  present 
challenges  to  the  sustainment  of  partnerships  with  the  K-12  community  (Williams,  2002).  In 
this  context,  participation  in  such  projects  as  the  MSP  is  likely  to  be  viewed  as  service  to  the 
institution  and  community.  Information  gathered  in  interviews  with  IHE  deans  and  depart¬ 
ment  administrators  was  consistent  with  this  view.  Participation  in  the  MSP,  without  any 


Table  6.2 

Factors  Associated  with  Partnership  Development  Between  Institutions 


Institutional  Factor 

Contributes  to  Partnership 
Development 

Inhibits  Partnership 
Development 

Academic  reward  system 

V 

V 

Campus  initiatives 

V 

History  of  outreach  or  collaboration 

V 

Leadership 

V 

External  factors 

V 

Faculty  stability  or  administrative  turnover 

V 

Resources  (including  size) 

V 
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additional  activities,  was  considered  to  be  service  even  at  institutions  that  had  relatively  broad 
definitions  of  scholarship  as  espoused  by  Ernest  Boyer’s  (1997)  model  of  scholarship.  Boyer 
proposed  an  expanded  definition  of  scholarship  among  the  professorate  that  was  based  on  dis¬ 
covery,  integration,  application,  and  teaching.  Within  this  framework,  he  argued  that  all  forms 
of  scholarship  should  be  recognized  and  rewarded  and  that  this  will  lead  to  more  personal¬ 
ized  and  flexible  criteria  for  gaining  tenure.  Many  of  the  deans  and  department  administrators 
whom  we  interviewed  described  a  variety  of  ways  in  which  MSP  participation,  in  conjunc¬ 
tion  with  other  activities,  could  count  toward  scholarship.  As  explained  by  one  dean,  “Doing 
workshops  and  [teacher  leader]  academies  and  working  with  teacher  fellows  is  service,  but 
presentations  at  [conferences]  is  considered  scholarship.”  Another  dean  clarified  the  distinction 
between  scholarship  and  service  in  this  manner: 

Scholarship  is  defined  as  when  you  have  taken  your  service  and  written  or  talked  about 
it  and  shared  the  experience  with  others.  If  I  did  a  content  [short  course],  that  would  be 
service.  If  I  wrote  an  article  about  how  I  developed  the  [course]  and  how  the  teachers  went 
back  and  used  that  information,  that  would  be  scholarship. 

Both  of  these  comments  suggest  that  using  knowledge  and  skills  drawn  from  the  experience  of 
MSP  participation  could  contribute  to  scholarship. 

On  the  other  hand,  some  department  administrators  perceived  MSP  participation  alone 
as  counting  for  more  than  just  service.  IHE  faculty  interviews  suggested  that  participation  in 
the  MSP  could  (and  should)  be  viewed  as  teaching  and  professional  development  in  addition 
to  service.  One  tenured  professor  explained,  “If  a  faculty  member  goes  to  a  seminar,  that’s  evi¬ 
dence  of  trying  to  improve  teaching.  Doing  a  TLA  should  be  considered  the  same  as  that.”  It 
was  not  clear  to  what  extent  deans  supported  this  view  that  MSP  participation  alone  contrib¬ 
utes  to  teaching.  One  described  being  open  to  the  idea: 

If  a  faculty  member  were  to  choose  to  demonstrate  in  their  teaching  portfolio  for  tenure, 
promotion,  or  annual  merit,  they  can  use  methods  learned  from  the  MSP. . . .  People  haven’t 
used  that.  If  you  develop  courses  and  curriculum  and  assessment  of  objective  outcomes  and 
show  pre-  and  post-MSP  data,  that  would  be  a  guarantee  of  the  person’s  promotion. 

Another  dean  voiced  a  similar  opinion:  “I’m  willing  to  make  the  case  that  it’s  also  countable  as 
professional  development  for  the  faculty  member  if  they  become  appropriately  engaged.”  These 
comments  suggest  that  more  would  be  required  for  MSP  participation  to  be  considered  teach¬ 
ing  or  professional  development.  And  yet,  MSP  participation  consumes  much  time  and  effort. 
Participating  in  workshops  and  TLAs  is  time-consuming,  and  absent  clear  signals  from  the 
administration  that  it  is  valued,  faculty  members  expressed  concern  about  the  amount  of  time 
they  should  devote  to  these  efforts.  Moreover,  some  faculty  members  can  get  credit  toward 
teaching  and  professional  development  and  service  in  other,  less  time-consuming  ways.  As  one 
faculty  member  explained, 

Most  people  can  get  an  A+  rating  on  teaching  and  service  without  participation  in  the  MSP. 

These  are  areas  that  are  easy  to  do  well  on,  but  it  is  research  that  is  the  ultimate  hurdle. 

Although  it  is  true  that  faculty  members  are  not  participating  in  the  MSP  to  get  credit 
toward  tenure  and  promotion,  the  time  commitment  required  by  the  MSP  was  viewed  as 
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costly  because  it  limited  the  amount  of  time  available  to  spend  on  the  research  activities  that 
are  weighted  heavily  in  tenure  and  promotion  decisions.  As  explained  by  one  midcareer  faculty 
member,  “It  cuts  into  time  spent  on  research  which  would  have  had  tangible  benefits.” 

Ambiguity  associated  with  how  MSP  participation  will  be  viewed  in  tenure  and  promo¬ 
tion  decisions,  juxtaposed  with  the  clarity  of  the  value  of  other  activities,  created  a  sense  among 
many  interviewees  that  MSP  participation  was  risky  for  junior  faculty  members.  For  example, 
in  some  IHEs,  the  number  of  publications  was  critical  for  promotion;  thus,  any  time  away 
from  writing  was  viewed  as  detrimental.  When  asked  about  challenges  to  sustained  participa¬ 
tion  in  the  MSP,  one  senior  faculty  member  stated  that  an  important  one  is  “the  time  issue  for 
faculty  who  are  not  tenured.  MSP  would  count  to  some  degree,  but  it  has  less  of  a  value  for 
tenure  than  research.  I  think  if  you  did  it  right  and  make  the  claim  that  MSP  is  helping  you  to 
be  a  better  teacher  it  could  work,  but  it’s  a  risky  thing.” 

Changes  to  IHE  Recognition  and  Reward  Policies 

As  indicated  by  the  comments  in  the  preceding  section,  changes  in  recognition  and  reward 
policies  may  be  important  in  sustaining  faculty  member  participation  in  the  MSP.  The  con¬ 
sensus  among  those  we  interviewed  was  that,  at  a  bare  minimum,  the  reward  system  should  be 
structured  so  that  MSP  participation  does  not  directly  or  inadvertently  count  against  promo¬ 
tion  and  tenure. 

From  the  start  of  the  MSP  in  2003,  there  has  been  considerable  effort  by  deans  to  forge 
relationships  with  one  another  and  articulate  their  support  of  the  MSP  and  its  sustainability. 
An  example  of  this  is  the  dean’s  dinners,  an  event  held  twice  per  year  in  which  deans  of  each 
IHE  partner  discuss  issues  related  to  the  MSP  and  their  sustained  participation.  One  outcome 
of  these  gatherings  was  a  joint  policy  statement  on  the  importance  of  faculty  involvement  in 
the  MSP,  which  was  issued  in  2006  by  the  chief  academic  officers  of  the  four  IHEs.  In  this 
statement,  the  chief  academic  officers  recognized  the  NSF’s  expectations  to  “promote  institu¬ 
tional  and  organizational  change  to  sustain  partnerships’  promising  practices  and  policies.” 
This  statement  affirms  that  effective  (supported  by  evidence)  work  by  faculty  in  MSP  activities 
should  be  included  in  their  applications  for  advancement  or  tenure.  Yet,  the  statement  respects 
the  individual  character  of  each  IHE  and  does  not  attempt  to  define  how  much  the  work 
should  count  toward  the  requirements  for  advancement  or  tenure.  It  is  significant  that  this 
statement  puts  on  record  that  each  IHE,  whether  through  its  rank  and  tenure  committee  or 
chief  academic  officer,  has  an  obligation  to  carefully  review  the  MSP  work  described  by  faculty 
members  in  their  applications  for  advancement  or  tenure  (Hipps,  2006-2007). 

As  a  result  of  these  actions,  some  institutions  appear  to  have  made  modest  progress  toward 
changing  their  recognition  and  reward  policies  to  support  MSP  participation.  One  dean  said, 

My  being  appointed  dean  has  been  very  helpful,  since  I  can  help  to  make  this  kind  of  activ¬ 
ity  count  for  something  towards  tenure.  I’m  encouraging  the  writing  up  of  outcomes  and 
results  of  MSP  activities  in  education  journals.  MSP  activities  are  clearly  seen  as  valuable  to 
the  college,  the  students,  and  the  local  community. 

Faculty  members  also  reported  some  success  in  having  policies  changed  so  that  MSP 
activities  would  be  considered  for  teaching  and,  potentially,  scholarship.  At  one  institution,  a 
faculty  member  reported,  “Yes,  the  rank  and  tenure  issues  have  changed.  The  science  and  math 
faculty  have  become  a  little  more  comfortable  with  participation-based  research.  Changes  are 
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occurring  on  the  rank  and  tenure  committee.”  At  this  institution,  it  appears  as  though  MSP 
activities  and  work  are  being  incorporated  into  the  academic  reward  system  and  that  STEM 
faculty  members  are  engaged  in  participation-based  research  on  the  MSP  or  their  teaching 
practices. 

In  the  IHEs  with  less  developed  partnerships,  as  indicated  by  fewer  reciprocally  described 
partnerships,  an  unsupportive  academic  reward  system  is  commonly  described.  This  suggests 
that  institutions  that  have  not  formally  changed  their  reward  and  recognition  policies  may 
need  to  consider  doing  so  in  order  to  sustain  MSP  partnerships  beyond  the  grant  period.  In 
the  past  decade,  many  institutions  and  accrediting  organizations  have  attempted  to  implement 
changes  based  on  Boyer’s  model  of  broader  scholarship  definition.  A  study  by  Massey,  Wilger, 
and  Colbeck  (2000),  based  on  interviews  with  more  than  300  faculty  members  from  a  variety 
of  institutions,  including  research  universities,  doctoral-granting  institutions,  and  liberal  arts 
colleges,  identified  certain  features  of  departments  that  maintain  a  supportive  culture  for  all 
forms  of  scholarship.  These  include  the  following: 

•  Faculty  interact  frequently,  fostering  a  healthy  awareness  of  and  respect  for  diverse  quali¬ 
ties  possessed  by  colleagues. 

•  There  is  a  sense  of  generational  equity;  senior  and  junior  faculty  are  viewed  as  equals. 

•  Faculty  value  student  evaluations  and  use  them  periodically  to  improve  curricula  and 
instruction. 

•  Incentives  for  research,  service,  and  quality  teaching  are  balanced. 

Thus,  a  balanced  focus  on  all  forms  of  scholarship  could  be  beneficial  beyond  sustaining 
the  MSP,  accruing  other  benefits  to  the  IHEs,  such  as  improving  collegiality  and  the  overall 
climate. 

Informal  Rewards  for  MSP  Participation 

In  addition  to  their  descriptions  of  formal  recognition  and  reward  policies,  faculty  members 
also  reported  receiving  informal  rewards  or  recognition  for  their  MSP  work.  Explaining  the 
benefits  of  participation  in  the  MSP,  one  faculty  member  said,  “We  got  validation  for  things  we 
were  doing  and  we  made  changes  to  other  things.”  Another  said,  “[The  institution  administra¬ 
tors]  recognize  us  for  it.  [Our  dean]  is  good  at  letting  administrators  know.” 

Additionally,  faculty  members  reported  that  participating  in  the  MSP  was  important  to 
them  as  scholars  and  educators.  One  said, 

I’m  really  enjoying  it.  It  fits  in  with  my  lifetime  goals.  The  best  part  of  it  is  that  it’s  long 
enough  to  really  develop  and  have  long-term  results.  This  has  enough  depth,  enough  initial 
funding,  enough  soldiers  in  the  army,  that  I  can  see  it  having  a  lasting  effect. 

Another  reported,  “[The  MSP  is]  something  exciting  and  new.  It’s  a  valuable  and  useful  thing, 
and  people  like  to  feel  valued  and  useful.” 

Thus,  it  appears  that  in  some  cases,  IHE  faculty  members  are  rewarded  both  formally 
and  informally.  Such  rewards  are  likely  to  motivate  their  participation  in  the  MSP  and  may 
be  necessary  to  sustain  partnerships  beyond  the  grant  period.  Continued  participation  of  fac¬ 
ulty  members  will  be  important  for  maintaining  the  high-quality  preparation  of  preservice 
teachers. 
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The  next  section  explores  how  revisions  to  IHE  courses  have  affected  not  only  the  content 
of  courses  for  preservice  teachers,  but  also  faculty  instructional  practices. 


Implementing  Challenging  Courses:  IHE  Course  Revisions 

In  2001,  the  state  of  Pennsylvania  mandated  that  teacher  certification  programs  certify  com¬ 
petency  to  teach  the  content  specified  in  the  K-12  standards.  This  mandate  created  an  impetus 
for  IHE  partners  to  refine  the  coursework  for  preservice  teachers  to  meet  these  new  expec¬ 
tations.  One  question  addressed  in  the  IHE  qualitative  analyses  is  how  MSP  participation 
influenced  these  courses,  particularly  their  content.  One  direct  influence  on  IHE  course  con¬ 
tent  is  the  TF  program.  As  discussed  in  Chapter  Two,  the  TF  program  enables  two  teachers 
from  each  district  to  spend  a  summer,  semester,  or  academic  year  at  a  partner  IHE.  During 
each  term  that  they  spend  as  a  TF,  the  teachers  work  with  faculty  members  to  help  revise  two 
courses  in  which  preservice  teachers  enroll.  They  also  take  a  math  or  science  college  course 
and  participate  in  other  MSP  activities.  Thus,  the  TF  experience  is  designed  to  contribute  to 
the  preparation  of  preservice  teachers  through  course  revisions,  to  strengthen  the  TF’s  content 
knowledge  in  math  or  science  through  IHE  course  enrollment,  and  to  help  build  partnerships 
through  participation  in  MSP  activities. 

The  MSP  project  set  a  benchmark  to  revise  131  courses  over  the  course  of  the  project. 
As  of  the  fifth  year,  97  courses  had  been  revised,  some  at  each  participating  IHE.  These  have 
included  biology,  chemistry,  physics,  education,  computer  science,  and  environmental  science 
courses.  We  gathered  information  on  the  revised  courses  from  individual  interviews  with  fac¬ 
ulty  members  and  TFs,  from  faculty  responses  on  an  MSP  course-revision  questionnaire,2  and 
through  focus  groups  with  former  TFs. 

The  character  of  the  course  revisions  varied  across  the  different  disciplines.  However,  they 
could  be  generally  categorized  as  activity-based,  content-based,  or  structure-based  revisions. 
Activity-based  course  revisions  appear  to  be  the  most  common.  Because  many  of  the  TFs  were 
experienced  in  leading  group  and  cooperative  learning  activities,  many  of  the  courses  were 
revised  to  include  more  group  projects  and  activities  to  increase  student  participation.  One  fac¬ 
ulty  member  remarked  that  the  greatest  value  of  the  course-revision  process  came  from  a  TF’s 
recommendations,  based  on  experience,  related  to  the  development  of  problem-based  coopera¬ 
tive  learning  activities. 

Content-based  revisions  were  less  common.  These  involved  augmenting  the  course  with 
additional  materials  to  encompass  a  wider  variety  of  approaches  to  presenting  the  content.  For 
example,  one  faculty  member  described  revising  a  mathematics  course  to  incorporate  math 
problem-solving  approaches  into  each  class  session  and  to  use  examples  from  the  Everyday 
Math  curriculum.3  In  other  cases,  content-based  revisions  incorporated  new  material  that 
was  targeted  directly  to  future  math  and  science  teachers.  Some  faculty  members  questioned 
whether  TFs  should  be  expected  to  help  revise  the  actual  material  to  be  taught.  As  one  faculty 
member  explained,  “The  fellows  can  help  by  knowing  what  the  standards  are  in  the  secondary 
schools,  what  assessment  tools  are  used  there.  They  can’t  necessarily  help  with  content.”  Yet,  the 


2  Course-revision  questionnaires  were  received  for  52  revised  courses.  Not  all  of  the  questionnaires  had  been  received  at 
the  writing  of  this  report. 

3  Everyday  Math  is  one  of  the  research-based  materials  promoted  by  the  MSP. 
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increased  curriculum  alignment  with  Pennsylvania  state  standards  was  considered  important, 
as  it  was  likely  to  create  a  more  seamless  transition  between  K-12  curricula  and  IHE  courses. 
One  faculty  member  describing  the  course  revision  stated,  “[Now  I  am]  mapping  standards  to 
the  syllabus  and  I  will  emphasize  the  standards  in  some  cases.  I’ll  try  to  implement  some  of  the 
ideas  with  the  standards.” 

Finally,  a  number  of  course  revisions  were  structure-based,  focusing  on  changes  to  the 
sequencing  of  topics  to  make  the  course  more  effective  for  student  learning.  For  example,  one 
of  the  course  revisions  included  reorganizing  lecture  topics  to  reflect  the  sequence  in  a  new 
textbook  and  the  incorporation  of  new  lecture  topics  in  the  course. 

Despite  the  variation  in  the  character  of  the  course  revisions,  faculty  members  were  fairly 
similar  in  terms  of  their  expectations  about  which  aspects  of  the  course  would  be  changed  as 
a  result  of  the  course  revision.  More  than  half  of  the  faculty  members  who  responded  to  the 
MSP  course-revision  questionnaire  expected  moderate  to  substantial  changes  to  their  in-class 
activities,  whereas  approximately  half  reported  “no  change”  when  asked  about  change  related 
to  assessments  of  learning  outside  the  classroom.  The  MSP  course-revision  questionnaire  also 
asked  faculty  members  to  describe  the  likely  impact  of  course  revisions  on  their  students’  learn¬ 
ing  experiences.  Most  responded  that  they  expected  their  students  to  be  more  engaged.  As  a 
result  of  the  revisions,  such  as  increased  use  of  cooperative  learning  or  inquiry-based  activities, 
faculty  members  said  that  they  expect  students  not  only  to  engage  more,  but  also  to  take  more 
responsibility  for  their  own  learning: 

Students  engage  this  course  somewhat  differently  because  they  are  no  longer  evaluated 
purely  on  the  basis  of  quiz  or  exam  scores.  Now  a  substantial  portion — 20  percent — of 
their  grade  comes  from  cooperative  learning  activities,  which  they  appear  to  enjoy. 

In  addition,  faculty  members  said  that  they  expect  the  revisions  to  create  a  much  richer  envi¬ 
ronment  for  student  learning,  enabling  students  to  increase  their  skills  in  such  areas  as  com¬ 
munication,  presentation,  and  hypothesis  development. 


Summary  of  Findings  Related  to  Sustainable  Partnership 

Development  of  sustainable  partnerships  is  an  important  goal,  not  just  for  the  MSP  but  also 
the  national  goals  of  the  NSF.  Program  announcements  describing  the  Math  and  Science  Part¬ 
nership  program  suggest  that  projects  should  be  partnership-driven.  This  implies  that  partner¬ 
ship  drives  the  NSF  initiative  and  that  both  universities  and  K-12  institutions  play  important 
roles  (Kingsley  and  Waschak,  2005).  In  many  ways,  the  third  goal  of  the  MSP,  the  creation  of 
sustainable  partnerships,  undergirds  the  achievement  of  the  other  two  project  goals.  The  extent 
to  which  the  MSP  has  achieved  sustainable  partnerships  is  unclear  at  this  stage,  but  there  are 
positive  indications  of  partnership  development  between  IHE  and  K-12  educators,  between 
IHEs,  and  between  departments  within  a  given  IHE.  Moreover,  one  of  the  important  fac¬ 
tors  associated  with  partnership  development,  the  faculty  reward  systems  of  IHEs,  has  shown 
modest  progress  toward  a  broader  definition  of  scholarship.  This  is  viewed  as  a  key  requirement 
for  maintaining  and  sustaining  faculty  involvement. 

An  important  measure  of  sustainable  partnership  is  the  number  of  courses  that  have 
been  revised  through  the  TF  program.  By  year  5,  approximately  three-fourths  of  the  targeted 
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131  courses  had  been  revised.  Responses  by  IHE  faculty  members  on  course-revision  question¬ 
naires  support  the  value  of  revising  courses.  In  particular,  the  revisions,  along  with  their  own 
participation  in  MSP  activities,  helped  faculty  members  embrace  a  wider  variety  of  approaches 
to  presenting  the  material  to  their  students.  As  a  result  of  these  changes,  faculty  members  were 
optimistic  that  students  would  become  more  engaged  in  coursework  and  take  responsibility 
for  their  own  learning.  As  the  MSP  continues,  following  the  progress  of  the  impact  of  course 
revisions  on  student  learning  may  become  an  additional  measure  for  tracking  the  progress  of 
sustainable  partnerships. 


CHAPTER  SEVEN 

Conclusion 


This  monograph  has  examined  MSP  progress  toward  meeting  its  three  goals  of  increasing  the 
mathematics  and  science  achievement  of  K-12  students,  increasing  the  quality  of  the  K— 16 
educator  workforce,  and  creating  sustainable  partnerships.  Analyses  relied  on  data  collected 
through  the  MSP’s  first  five  years  of  implementation,  from  surveys  of  teachers  and  principals, 
qualitative  data  from  participating  IHEs,  educator  participation  records,  and  student  achieve¬ 
ment.  Here,  we  briefly  summarize  the  findings  reported  in  earlier  chapters. 

Achievement  analyses  found  that  MSP  school  districts  experienced  trends  of  increase  in 
student  mathematics  and  science  scores  during  the  course  of  the  project.  For  mathematics, 
similar  trends  were  observed  throughout  Pennsylvania;  for  science,  there  was  no  external  ref¬ 
erence  for  comparison.  Further  analyses  examined  the  relationship  between  educators’  MSP 
participation  and  students’  math  and  science  achievement.  MSP  participation  measures  were 
developed  to  account  for  differences  in  the  potential  impact  of  educators  who  play  a  leadership 
role,  as  opposed  to  educators  who  teach  but  do  not  play  a  leadership  role.  These  measures  were 
then  used  in  three  distinct  analytic  strategies  for  statistically  modeling  the  relationship  between 
educator  participation  and  student  achievement.  Results  of  these  analyses  showed  only  a  few 
significant  relationships  between  MSP  participation  by  educators  and  student  math  achieve¬ 
ment.  These  significant  relationships  appeared  among  many  nonsignificant  findings  and  were 
inconsistent  across  cohorts  and  analyses.  As  a  result,  they  do  not  enable  us  to  draw  any  overall 
conclusions  about  the  effects  of  the  MSP  on  changes  in  math  or  science  achievement. 

Analyses  of  K-12  survey  and  IHE  qualitative  data  examined  evidence  of  change  in  mid¬ 
term  outcomes  related  to  the  quality  of  the  K-16  educator  workforce.  Generally,  educators  at 
both  the  K-12  and  IHE  levels  reported  changes  in  leadership  and  instruction  that  are  con¬ 
sistent  with  the  MSP  theory  of  action.  K-12  teachers  reported  that  MSP  activities  increased 
their  awareness  and  understanding  of  math  and  science  concepts  and  how  students  think 
about  math  and  science  and  helped  them  change  their  teaching  practices.  They  also  reported 
that  the  professional  development  that  they  received  was  more  often  relevant  to  needs  and 
was  focused  on  instructional  approaches  and  individual  student  learning.  Principals  reported 
that  the  principal  seminars  influenced  their  views  and  behaviors  as  principals.  IHE  faculty 
reported  a  greater  emphasis  on  student- centered  instruction  and  more  awareness  of  different 
pedagogical  techniques.  These  findings  suggest  that  the  activities  of  the  MSP  may  be  having 
the  intended  effect  on  midterm  outcomes  associated  with  increasing  the  quality  of  the  educa¬ 
tor  workforce.  However,  further  statistical  analyses  of  the  relationship  of  participation  in  MSP 
activities  to  survey  scales  tracking  key  midterm  outcomes  did  not  provide  evidence  that  the 
MSP  is  responsible  for  the  changes  reported  by  educators.  Data  from  the  K-12  case  studies 
may  be  helpful  in  interpreting  these  findings. 
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IHE  qualitative  analyses  examined  the  development  of  sustainable  partnerships  in  the 
MSP  and  found  positive  indications  of  partnership  development  between  IHE  and  K-12  edu¬ 
cators,  between  IHEs,  and  between  departments  within  IHEs.  Moreover,  the  analyses  found 
modest  progress  toward  a  broader  definition  of  scholarship  in  IHE  faculty  reward  systems, 
an  important  factor  in  partnership  development.  Participation  in  MSP  activities,  along  with 
revisions  to  IHE  courses  through  the  TF  program,  helped  faculty  embrace  a  wider  variety  of 
approaches  to  presenting  the  material  to  their  students.  As  a  result  of  these  changes,  faculty 
members  were  optimistic  that  students  would  become  more  engaged  in  coursework  and  take 
responsibility  for  their  own  learning. 

In  sum,  the  evaluation  found  numerous  indications  that  changes  are  occurring  that  are 
consistent  with  the  MSP  theory  of  action.  Thus,  MSP  partners  appear  to  be  making  progress 
toward  the  three  MSP  goals,  though  attempts  to  statistically  link  this  progress  to  MSP  partici¬ 
pation  were  not  successful.  There  are  several  limitations  to  note.  The  evaluation  was  designed 
to  be  selective  in  its  data  collection  and  analyses,  primarily  assessing  the  project’s  achievement 
of  its  goals  and  the  major  pathways  toward  achieving  those  goals;  the  evaluation  relies  on  self- 
report  data  from  a  sample  of  participants  and  is  subject  to  common  potential  biases  associated 
with  such  data;  and,  importantly,  if  the  MSP  intervention  strategies  require  more  than  four 
years  to  achieve  project  goals,  this  evaluation  may  not  detect  the  impact.  This  monograph 
concludes  the  MSP  evaluation  activities  of  RAND  Education.  Note,  however,  that  it  does  not 
constitute  a  final  evaluation  of  the  MSP.  The  project,  including  data  collection  and  evaluation 
activities,  is  expected  to  continue  through  a  forecasted  end  date  in  2010. 


APPENDIX  A 


Achievement  Analysis  Methods  and  Results 


This  appendix  contains  additional  information  related  to  the  achievement  analyses  discussed 
in  Chapter  Four. 


Student  Achievement  Analysis  Methods 

Student-Level  Achievement  Models 

For  each  cohort  analyzed  in  the  student-level  achievement  models,  a  multilevel,  mixed-effect, 
cross- classified  model  was  used  to  predict  the  post-test  scores  using  the  pretest  scores,  demo¬ 
graphic  variables  (race/ethnicity,  gender,  and  socioeconomic  status),  and  individual-level  par¬ 
ticipation  and  teaching  experience  of  linked  educators  in  each  year  as  covariates  and  accounting 
for  the  clustering  of  students  within  classrooms  (Lockwood,  et  ah,  2007;  McCaffrey,  Lock- 
wood,  et  ah,  2004).  The  model  used  for  analysis  was 

m  m  m 

y, = ft +*TxT  +E/W + le„ +Xr  A +«,. 

k=\  k=\  k=\ 


where  Y.  denotes  student  i  s  standardized  post-MSP  math  achievement  score,  /i(.  is  the  over¬ 
all  average  math  achievement,  X T  is  a  vector  of  student  demographic  information  and  prior 
math  and  reading  achievement  scores,  A  is  a  vector  of  estimated  coefficients  for  the  variables 
included  in  X 1 ,  k  represents  the  year  of  MSP,  and  m  refers  to  the  length  of  MSP  exposure  for 
educators  linked  to  a  certain  cohort  of  students.  The  variable  m  differs  across  cohorts  of  stu¬ 
dents.  It  is  equal  to  3  for  cohort  G  (grade  11),  2  for  cohort  K  (grade  7),  and  1  for  cohorts  L 
(grade  6)  and  N  (grade  4).1  P.k  represents  the  average  of  individual-level  participation  across  all 
educators  linked  to  student  i  in  year  k.  /f  is  the  estimated  fixed-effect  coefficient  for  educators’ 
participation  in  year  k.  9k  indicates  the  residual  random  teacher  effect  to  account  for  clustering 
within  classrooms  in  year  k.  E.,  is  the  average  teaching  experience  across  all  educators  linked 
to  student  i  in  year  k.  y,  represents  the  estimated  fixed-effect  coefficient  for  educators’  teaching 
experience  in  year  k.  The  term  £  is  the  residual  error  term  with  mean  zero  and  variance  O1  £ . 
Estimation  of  these  coefficients  was  implemented  in  WinBUGS  using  Markov  chain  Monte 
Carlo  sampling  algorithms  (Spiegelhalter,  Thomas,  and  Best,  1999). 


1  Cohorts  J  and  M  were  not  included  in  the  student-level  analysis  due  to  the  limited  number  of  students  linked  to  educa¬ 
tors  in  all  years  between  pre-  and  post-MSP  math  achievement  tests. 
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Assumptions  underlying  this  model  include  the  following:  (1)  when  multiple  teachers  are 
linked  to  a  student  in  a  single  school  year,  information  about  the  percentage  of  time  a  student 
spent  with  each  of  them  is  not  available,  so  it  is  assumed  that  the  teachers  contributed  equally 
to  the  student’s  mathematics  achievement,  and  (2)  each  teacher’s  benefit  per  hour  of  participa¬ 
tion  is  assumed  to  be  constant.  Student-level  achievement  analyses  were  conducted  only  for  the 
mathematics  achievement  scores  because  the  PROM/SE  assessment  does  not  produce  compa¬ 
rable  student-level  scores. 

As  discussed  in  Chapter  Four,  the  students,  teachers,  schools,  and  districts  included 
in  the  student-level  analyses  were  constrained  by  data  limitations.  Figures  A.l  through  A.3 
show  the  participation  levels  of  leaders,  mathematics  teachers,  and  science  teachers,  respectively, 
in  the  districts  that  are  represented  in  the  student-level  analyses.  The  figures  show  that  all  levels 
of  district-level  participation  are  represented.  Figures  A.4  and  A.5  show  similar  information  for 
the  participation  of  mathematics  and  science  teachers,  respectively,  in  the  represented  schools. 
All  levels  of  school-level  participation  are  represented,  except  for  schools  with  the  highest  levels 
of  science  teacher  participation.  (As  mentioned  earlier,  these  models  do  not  examine  science 
achievement.) 

District-Level  Achievement  Models 

In  the  district-level  achievement  analyses,  a  linear  regression  model  was  used  to  predict  district- 
level  aggregated  math  achievement,  controlling  for  prior  district-level  aggregated  math  and 
reading  achievement;  the  ratios  of  female,  minority,  and  economically  disadvantaged  students; 
district-level  leaders’  total  participation;  and  adjusted  teacher  participation.  Because  leaders’ 

Figure  A.l 

Representativeness  of  Districts  Included  in  Student-Level  Models  in  Terms  of  Leader 
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Figure  A. 2 

Representativeness  of  Districts  Included  in  Student-Level  Models  in  Terms  of  Mathematics 
Teacher  Participation 
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Figure  A. 3 

Representativeness  of  Districts  Included  in  Student-Level  Models  in  Terms  of  Science 
Teacher  Participation 
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Figure  A. 4 

Representativeness  of  Schools  Included  in  Student-Level  Models  in  Terms  of  Mathematics 
Teacher  Participation 
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Figure  A. 5 

Representativeness  of  Schools  Included  in  Student-Level  Models  in  Terms  of  Science 
Teacher  Participation 


RAND  MG857-A.5 


Achievement  Analysis  Methods  and  Results  73 


participation  in  math  and  science  might  affect  teaching  and  learning  in  both  subject  areas,  the 
sum  of  leaders’  participation  hours  in  math  and  science  was  included  in  the  regression  model. 
Moreover,  the  adjusted  teacher  participation  in  math  or  science  was  used  in  the  corresponding 
regression  model.  The  model  used  for  analysis  was  as  follows: 

Y  =U+  kTXT  +[3rPr+  /3P  +  £, 

where  Yd_  refers  to  the  district-level  aggregated  achievement  score;  jJ.Q  is  the  overall  mean  for 
the  district-level  aggregated  achievement  score;  X  ris  a  vector  of  district-level  prior  achievement 
scores  and  ratios  of  female,  minority,  and  economically  disadvantaged  students;  Ar  contains 
estimated  coefficients  for  each  component  in  vector  XT ;  P  and  PT  are  district-level  leaders’ 
total  participation  and  adjusted  teachers’  participation,  respectively;  j3L  and  /jy  are  estimated 
coefficients  for  leaders’  total  participation  and  adjusted  teachers’  participation,  respectively; 
and  £  is  the  residual  error  item  with  mean  zero  and  variance  O2  £  . 

These  district-level  achievement  analyses  were  conducted  for  both  mathematics  and  sci¬ 
ence  achievement  scores.  For  each  cohort  in  the  analysis,  variables  for  district-level  leaders’ 
total  participation  and  adjusted  teachers’  participation  contain  the  participation  of  the  corre¬ 
sponding  educators  from  the  beginning  of  MSP  implementation  up  to  the  year  of  the  post-test. 
Within  the  cohorts  and  districts  contributing  data  to  these  analyses,  the  correlation  of  leaders’ 
and  teachers’  participation  is  high — in  the  range  of  0.8  to  0.9.  Therefore,  the  measures  were 
included  in  models  both  individually  and  simultaneously;  the  results  were  not  sensitive  to  this 
choice.  For  district-level  analyses  of  MSP  participation  and  science  achievement  for  cohorts  FI 
and  K,  students’  science  achievement  scores  in  2003-04  were  used  as  covariates  in  the  regres¬ 
sion  model.  For  the  other  four  cohorts,  students’  prior  math  and  reading  achievement  scores 
were  used  because  no  science  pretest  score  was  available.  This  model  was  implemented  in  Stata® 
version  10  (StataCorp,  2007). 

Statewide  Comparison-Group  Achievement  Models 

For  the  statewide  achievement  models,  the  three  comparison  groups  described  in  Chapter  Four 
were  formed  using  the  method  described  by  McCaffrey,  Ridgeway,  and  Morral  (2004).  The 
method  uses  weights  to  reduce  pretreatment  differences  between  the  MSP  and  comparison 
districts.  Tables  A.37  through  A.39,  later  in  this  appendix,  show  the  balance  in  pretreatment 
covariates  before  and  after  matching  for  the  three  comparison  groups.  The  propensity  weights 
were  calculated  in  R  (R  Development  Core  Team,  undated)  using  the  package  twang  (Ridge¬ 
way,  McCaffrey,  and  Morral,  2006). 

The  analysis  uses  the  percentage  of  students  scoring  at  the  advanced  or  proficient  level  on 
the  mathematics  PSSA.  Separate  models  were  run  for  each  cohort  of  students  for  whom  PSSA 
mathematics  proficiency  levels  were  available  both  in  2003-04  or  earlier  (as  a  pre-MSP  mea¬ 
sure)  and  in  2006-07  (as  a  post-MSP  measure).  The  model  used  for  analysis  was  as  follows: 

Y.  =u+  XTXT  +  BP  +  £, 

where  Y dfM  refers  to  district-level  aggregate  proficiency  levels,  jjfi  is  the  overall  mean  for 
district-level  aggregated  proficiency,  XT  is  a  vector  of  district-level  pre-MSP  proficiency  levels 
and  variables  that  were  not  well  balanced  in  the  propensity  match,  XT  contains  estimated  coef- 
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ficients  for  each  component  in  vector  XT,  and  P  is  district-level  value  for  leaders’  total  partici¬ 
pation  or  teachers’  adjusted  participation.  Of  primary  interest  is  /?,  the  estimated  coefficient 
for  participation.  £  is  the  residual  error  item  with  mean  zero  and  variance  G2£.  The  models 
were  run  separately  on  the  three  comparison  groups  and  produced  generally  consistent  results. 
The  models  were  fit  in  R  using  svyglm  (from  the  survey  package;  see  Lumley,  2004),  a  gen¬ 
eralized  linear  model  with  inverse-probability  weighting  and  standard  errors  that  account  for 
weighting. 


Student-Level  Analysis  Results  for  Mathematics 

Tables  A.l  through  A.  8  present  the  student-level  results  for  mathematics  for  each  cohort.  In 
the  tables  of  descriptive  statistics,  N  is  the  number  of  students  and  N  is  the  number  of  teach¬ 
ers.  All  test  scores  are  standardized.  Female,  minority,  individualized  education  plan  (IEP), 
and  disadvantaged  students  represent  the  proportions  of  those  students  in  a  district.  For  each 
student,  teaching  experience  in  a  particular  school  year  is  the  average  of  teaching  experience 
of  all  teachers  linked  to  that  student  in  that  year;  teacher  participation  in  a  particular  school 
year  is  the  average  of  log-transformed  participation  hours  of  all  teachers  linked  to  that  student 
in  that  year. 
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Table  A.1 

Descriptive  Statistics  of  Variables  Included  in  the  Student-Level  Mixed-Effect  Model  Analysis  for 
Mathematics,  Cohort  G  (Grade  11  in  2006-07) 


Variable 

NS 

N, 

Mean 

Std.  Dev. 

Min. 

Max. 

Math  score  in  2006-07 

1,944 

0.00 

1.00 

-2.57 

4.03 

Math  score  in  2003-04 

1,944 

0.00 

1.00 

-7.06 

4.52 

Reading  score  in  2003-04 

1,944 

0.00 

1.00 

-7.16 

3.75 

Female 

1,944 

0.49 

0.50 

0.00 

1.00 

Minority 

1,944 

0.07 

0.26 

0.00 

1.00 

IEP 

1,944 

0.05 

0.23 

0.00 

1.00 

Disadvantaged  students 

1,944 

0.17 

0.37 

0.00 

1.00 

Teaching  experience  of  teachers  in  2004-05 

1,944 

94 

13.39 

9.68 

0.41 

36.00 

Teaching  experience  of  teachers  in  2005-06 

1,944 

116 

11.61 

8.51 

1.00 

37.00 

Teaching  experience  of  teachers  in  2006-07 

1,944 

115 

10.38 

7.10 

0.00 

35.00 

Teacher  participation  in  2004-05 

1,944 

94 

2.42 

1.59 

0.00 

4.83 

Teacher  participation  in  2005-06 

1,944 

116 

2.98 

1.48 

0.00 

5.36 

Teacher  participation  in  2006-07 

1,944 

115 

3.00 

1.69 

0.00 

5.41 

Table  A. 2 

Student-Level  Mixed-Effect  Model  Analysis  Results  for  Mathematics, 

Cohort  G  (Grade  11  in  2006-07) 

Variable 

Coefficient 

Std.  Err. 

95%  Confidence 
Interval  of  the 
Coefficient 

z 

p-value 

Math  score  in  2003-04 

0.59 

0.02 

0.56 

0.63 

27.65 

0.00 

Reading  score  in  2003-04 

0.09 

0.02 

0.06 

0.13 

5.09 

0.00 

Female 

0.15 

0.02 

0.11 

0.19 

6.32 

0.00 

Minority 

-0.04 

0.05 

-0.12 

0.03 

-0.91 

0.36 

IEP 

-0.11 

0.06 

-0.22 

-0.01 

-1.83 

0.07 

Disadvantaged  students 

-0.03 

0.03 

-0.09 

0.02 

-0.91 

0.37 

Teaching  experience  of  teachers  in  2004-05 

0.00 

0.00 

0.00 

0.01 

1.68 

0.09 

Teaching  experience  of  teachers  in  2005-06 

0.01 

0.00 

0.00 

0.01 

2.52 

0.01 

Teaching  experience  of  teachers  in  2006-07 

0.01 

0.00 

0.00 

0.01 

2.43 

0.02 

Teacher  participation  in  2004-05 

-0.01 

0.02 

-0.04 

0.02 

-0.36 

0.72 

Teacher  participation  in  2005-06 

-0.01 

0.02 

-0.04 

0.02 

-0.40 

0.69 

Teacher  participation  in  2006-07 

-0.01 

0.02 

-0.04 

0.02 

-0.56 

0.57 
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Table  A. 3 

Descriptive  Statistics  of  Variables  Included  in  the  Student-Level  Mixed-Effect  Model  Analysis  for 
Mathematics,  Cohort  K  (Grade  7  in  2006-07) 


Variable 

NS 

N, 

Mean 

Std.  Dev. 

Min. 

Max. 

Math  score  in  2006-07 

1,978 

0.00 

1.00 

-2.65 

4.44 

Math  score  in  2004-05 

1,978 

0.01 

1.00 

-3.18 

3.91 

Reading  score  in  2004-05 

1,978 

0.01 

1.00 

-6.38 

2.94 

Female 

1,978 

0.52 

0.50 

0.00 

1.00 

Minority 

1,978 

0.12 

0.32 

0.00 

1.00 

IEP 

1,978 

0.08 

0.28 

0.00 

1.00 

Disadvantaged  students 

1,978 

0.32 

0.47 

0.00 

1.00 

Teaching  experience  of  teachers  in  2005-06 

1,978 

64 

16.42 

11.35 

1.00 

34.00 

Teaching  experience  of  teachers  in  2006-07 

1,978 

58 

10.76 

8.46 

0.00 

34.00 

Teacher  participation  in  2005-06 

1,978 

64 

3.07 

1.33 

0.00 

5.24 

Teacher  participation  in  2006-07 

1,978 

58 

3.69 

1.04 

0.00 

5.36 

Table  A. 4 

Student-Level  Mixed-Effect  Model  Analysis  Results  for  Mathematics, 

Cohort  K  (Grade  7  in  2006-07) 

Variable 

Coefficient 

Std.  Err. 

95%  Confidence 
Interval  of  the 
Coefficient 

z 

p-value 

Math  score  in  2004-05 

0.60 

0.02 

0.57 

0.63 

33.22 

0.00 

Reading  score  in  2004-05 

0.13 

0.02 

0.10 

0.16 

7.42 

0.00 

Female 

0.00 

0.02 

-0.04 

0.04 

0.17 

0.87 

Minority 

0.00 

0.05 

-0.07 

0.08 

0.03 

0.98 

IEP 

-0.35 

0.05 

-0.43 

-0.27 

-7.04 

0.00 

Disadvantaged  students 

-0.10 

0.03 

-0.15 

-0.05 

-3.43 

0.00 

Teaching  experience  of  teachers  in  2005-06 

0.00 

0.00 

-0.01 

0.00 

-0.83 

0.40 

Teaching  experience  of  teachers  in  2006-07 

0.00 

0.01 

-0.01 

0.01 

-0.33 

0.74 

Teacher  participation  in  2005-06 

0.07 

0.02 

0.04 

0.10 

3.64 

0.00 

Teacher  participation  in  2006-07 

-0.03 

0.04 

-0.10 

0.03 

-0.81 

0.42 
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Table  A.5 

Descriptive  Statistics  of  Variables  Included  in  the  Student-Level  Mixed-Effect  Model  Analysis  for 
Mathematics,  Cohort  L  (Grade  6  in  2006-07) 


Variable 

NS 

N, 

Mean 

Std.  Dev. 

Min. 

Max. 

Math  score  in  2006-07 

4,105 

0.00 

1.00 

-2.80 

4.15 

Math  score  in  2005-06 

4,105 

0.00 

1.00 

-3.17 

3.67 

Reading  score  in  2005-06 

4,105 

0.00 

1.00 

-3.18 

4.10 

Female 

4,105 

0.50 

0.50 

0.00 

1.00 

Minority 

4,105 

0.16 

0.54 

0.00 

4.00 

IEP 

4,105 

0.11 

0.32 

0.00 

1.00 

Disadvantaged  students 

4,105 

0.27 

0.44 

0.00 

1.00 

Teaching  experience  of  teachers  in  2006-07 

4,105 

94 

11.23 

10.07 

0.00 

35.00 

Teacher  participation  in  2006-07 

4,105 

94 

2.82 

1.32 

0.00 

5.20 

Table  A. 6 

Student-Level  Mixed-Effect  Model  Analysis  Results  for  Mathematics, 

Cohort  L  (Grade  6  in  2006-07) 

Variable 

Coefficient 

Std.  Err. 

95%  Confidence 
Interval  of  the 
Coefficient 

z 

p-value 

Math  score  in  2005-06 

0.64 

0.01 

0.61 

0.66 

52.86 

0.00 

Reading  score  in  2005-06 

0.19 

0.01 

0.16 

0.21 

14.93 

0.00 

Female 

0.02 

0.02 

-0.01 

0.05 

1.33 

0.18 

Minority 

0.02 

0.02 

-0.01 

0.05 

1.53 

0.13 

IEP 

-0.08 

0.03 

-0.14 

-0.03 

-2.85 

0.00 

Disadvantaged  students 

-0.08 

0.02 

-0.12 

-0.04 

-3.94 

0.00 

Teaching  experience  of  teachers  in  2006-07 

0.00 

0.00 

0.00 

0.01 

1.18 

0.24 

Teacher  participation  in  2006-07 

-0.01 

0.01 

-0.03 

0.02 

-0.44 

0.66 
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Table  A.7 

Descriptive  Statistics  of  Variables  Included  in  the  Student-Level  Mixed-Effect  Model  Analysis  for 
Mathematics,  Cohort  N  (Grade  4  in  2006-07) 


Variable 

NS 

N, 

Mean 

Std.  Dev. 

Min. 

Max. 

Math  score  in  2006-07 

1,842 

0.00 

1.00 

-2.77 

4.34 

Math  score  in  2005-06 

1,842 

0.00 

1.00 

-3.01 

2.41 

Reading  score  in  2005-06 

1,842 

0.00 

1.00 

-3.18 

2.96 

Female 

1,842 

0.51 

0.50 

0.00 

1.00 

Minority 

1,842 

0.11 

0.31 

0.00 

1.00 

IEP 

1,842 

0.14 

0.35 

0.00 

1.00 

Disadvantaged  students 

1,842 

0.30 

0.46 

0.00 

1.00 

Teaching  experience  of  teachers  in  2006-07 

1,842 

156 

15.19 

11.02 

0.00 

37.00 

Teachers'  participation  in  2006-07 

1,842 

156 

2.66 

1.29 

0.00 

5.28 

Table  A. 8 

Student-Level  Mixed-Effect  Model  Analysis  Results  for  Mathematics, 

Cohort  N  (Grade  4  in  2006-07) 

Variable 

Coefficient 

Std.  Err. 

95%  Confidence 
Interval  of  the 
Coefficient 

z 

p-value 

Math  score  in  2005-06 

0.50 

0.02 

0.46 

0.53 

26.15 

0.00 

Reading  score  in  2005-06 

0.28 

0.02 

0.24 

0.32 

14.12 

0.00 

Female 

0.08 

0.03 

0.03 

0.14 

2.85 

0.00 

Minority 

-0.05 

0.05 

-0.16 

0.05 

-1.03 

0.30 

IEP 

-0.11 

0.04 

-0.20 

-0.03 

-2.56 

0.01 

Disadvantaged  students 

-0.12 

0.04 

-0.19 

-0.05 

-3.29 

0.00 

Teaching  experience  of  teachers  in  2006-07 

0.00 

0.00 

-0.01 

0.00 

-0.60 

0.55 

Teachers'  participation  in  2006-07 

0.03 

0.02 

0.00 

0.07 

1.69 

0.09 
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District- Level  Analysis  Results  for  Mathematics 

Tables  A.9  through  A.20  present  the  district-level  results  for  mathematics  for  each  cohort. 
All  test  scores  are  standardized.  Female,  minority,  IEP,  and  disadvantaged  students  represent 
the  proportions  of  those  students  in  a  district.  Leader  participation  is  the  sum  over  the  indi¬ 
cated  years  of  district  leadership  participation;  teacher  participation  is  the  sum  over  the 
indicated  years  of  adjusted  district-level  mathematics  teacher  participation.  These  measures  are 
described  in  Chapter  Four. 


Table  A.9 

Descriptive  Statistics  of  Variables  Included  in  District-Level  Regression  Analysis  for  Mathematics, 
Cohort  G  (Grade  11  in  2006-07) 


Variable 

N 

Mean 

Std.  Dev. 

Min. 

Max. 

Math  score  in  2006-07 

39 

-0.06 

0.41 

-1.17 

0.73 

Math  score  in  2003-04 

39 

-0.07 

0.40 

-0.96 

0.70 

Reading  score  in  2003-04 

39 

-0.06 

0.37 

-0.98 

0.56 

Female 

39 

0.50 

0.05 

0.33 

0.57 

Minority 

39 

0.16 

0.23 

0.00 

0.82 

IEP 

39 

0.13 

0.07 

0.03 

0.45 

Disadvantaged  students 

39 

0.31 

0.25 

0.01 

1.00 

Leader  participation,  2003-04  to  2006-07 

39 

1,427 

619 

92 

2,384 

Teacher  participation,  2003-04  to  2006-07 

39 

21.63 

12.76 

1.39 

48.60 

Table  A.10 

District-Level  Regression  Results  for  Mathematics,  Cohort  G  (Grade  11  in  2006- 

07) 

Variable 

Coefficient 

Std.  Err. 

95%  Confidence 
Interval  of  the 
Coefficient 

t 

p-value 

Math  score  in  2003-04 

0.61 

0.16 

0.30 

0.92 

3.84 

0.00 

Reading  score  in  2003-04 

0.19 

0.17 

-0.14 

0.52 

1.14 

0.26 

Female 

-1.39 

0.78 

-2.93 

0.14 

-1.78 

0.09 

Minority 

0.27 

0.27 

-0.26 

0.80 

1.00 

0.33 

IEP 

-0.29 

0.67 

-1.60 

1.02 

-0.43 

0.67 

Disadvantaged  students 

-0.71 

0.29 

-1.29 

-0.14 

-2.44 

0.02 

Leader  participation,  2003-04  to  2006-07 

0.00 

0.00 

0.00 

0.00 

-0.52 

0.60 

Teacher  participation,  2003-04  to  2006-07 

0.00 

0.00 

-0.01 

0.00 

-0.28 

0.78 
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Table  A.11 

Descriptive  Statistics  of  Variables  Included  in  District-Level  Regression  Analysis  for  Mathematics, 
Cohort  J  (Grade  8  in  2006-07) 


Variable 

N 

Mean 

Std.  Dev. 

Min. 

Max. 

Math  score  in  2006-07 

38 

-0.04 

0.44 

-1.10 

0.76 

Math  score  in  2003-04 

38 

0.00 

0.38 

-1.09 

0.62 

Reading  score  in  2003-04 

38 

-0.01 

0.38 

-1.23 

0.62 

Female 

38 

0.51 

0.05 

0.40 

0.60 

Minority 

38 

0.19 

0.26 

0.00 

0.90 

IEP 

38 

0.14 

0.05 

0.05 

0.31 

Disadvantaged  students 

38 

0.40 

0.26 

0.07 

1.00 

Leader  participation,  2003-04  to  2006-07 

38 

1,441 

621 

92 

2,384 

Teacher  participation,  2003-04  to  2006-07 

38 

21.82 

12.87 

1.39 

48.60 

Table  A.12 

District-Level  Regression  Results  for  Mathematics, 

Cohort  J  (Grade  8  in  2006-07) 

Variable 

Coefficient  Std.  Err. 

95%  Confidence 
Interval  of  the 
Coefficient 

t 

p-value 

Math  score  in  2003-04 

0.49 

0.24 

0.02  0.95 

2.04 

0.05 

Reading  score  in  2003-04 

0.03 

0.29 

-0.53  0.59 

0.11 

0.91 

Female 

0.13 

0.94 

-1.71  1.97 

0.14 

0.89 

Minority 

0.21 

0.32 

-0.41  0.83 

0.65 

0.52 

IEP 

-0.54 

1.16 

-2.81  1.73 

-0.47 

0.64 

Disadvantaged  students 

-1.11 

0.24 

-1.57  -0.65 

-4.69 

0.00 

Leader  participation,  2003-04  to  2006-07 

0.00 

0.00 

0.00  0.00 

0.85 

0.40 

Teacher  participation,  2003-04  to  2006-07 

-0.01 

0.00 

-0.01  0.00 

-1.74 

0.09 
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Table  A.13 

Descriptive  Statistics  of  Variables  Included  in  District-Level  Regression  Analysis  for  Mathematics, 
Cohort  K  (Grade  7  in  2006-07) 


Variable 

N 

Mean 

Std.  Dev. 

Min. 

Max. 

Math  score  in  2006-07 

39 

-0.08 

0.47 

-1.21 

0.89 

Math  score  in  2004-05 

39 

0.03 

0.32 

-0.89 

0.71 

Reading  score  in  2004-05 

39 

-0.01 

0.36 

-1.16 

0.57 

Female 

39 

0.53 

0.04 

0.46 

0.61 

Minority 

39 

0.20 

0.26 

0.00 

0.91 

IEP 

39 

0.15 

0.05 

0.05 

0.29 

Disadvantaged  students 

39 

0.41 

0.26 

0.06 

1.00 

Leader  participation,  2003-04  to  2006-07 

39 

1,427 

619 

92 

2,384 

Teacher  participation,  2003-04  to  2006-07 

39 

21.63 

12.76 

1.39 

48.60 

Table  A.14 

District-Level  Regression  Results  for  Mathematics, 

Cohort  K  (Grade  7  in  2006-07) 

Variable 

Coefficient  Std.  Err. 

95%  Confidence 
Interval  of  the 
Coefficient 

t 

p-value 

Math  score  in  2004-05 

0.51 

0.18 

0.16  0.85 

2.86 

0.01 

Reading  score  in  2004-05 

0.03 

0.28 

-0.51  0.57 

0.12 

0.91 

Female 

-0.01 

0.80 

-1.58  1.55 

-0.02 

0.99 

Minority 

-0.04 

0.24 

-0.52  0.44 

-0.15 

0.89 

IEP 

-0.37 

0.71 

-1.77  1.03 

-0.52 

0.61 

Disadvantaged  students 

-1.10 

0.28 

-1.64  -0.56 

-3.98 

0.00 

Leader  participation,  2003-04  to  2006-07 

0.00 

0.00 

0.00  0.00 

0.58 

0.56 

Teacher  participation,  2003-04  to  2006-07 

-0.01 

0.00 

-0.01  0.00 

-1.97 

0.06 
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Table  A.15 

Descriptive  Statistics  of  Variables  Included  in  District-Level  Regression  Analysis  for  Mathematics, 
Cohort  L  (Grade  6  in  2006-07) 


Variable 

N 

Mean 

Std.  Dev. 

Min. 

Max. 

Math  score  in  2006-07 

44 

-0.05 

0.45 

-1.06 

1.07 

Math  score  in  2005-06 

44 

-0.04 

0.39 

-0.91 

0.85 

Reading  score  in  2005-06 

44 

-0.07 

0.41 

-1.18 

0.53 

Female 

44 

0.51 

0.04 

0.40 

0.63 

Minority 

44 

0.19 

0.25 

0.00 

0.92 

IEP 

44 

0.14 

0.04 

0.06 

0.23 

Disadvantaged  students 

44 

0.40 

0.25 

0.08 

1.00 

Leader  participation,  2003-04  to  2006-07 

44 

1,425 

608 

92 

2,384 

Teacher  participation,  2003-04  to  2006-07 

44 

20.90 

12.78 

1.39 

48.60 

Table  A.16 

District-Level  Regression  Results  for  Mathematics, 

Cohort  L  (Grade  6  in  2006-07) 

Variable 

Coefficient  Std.  Err. 

95%  Confidence 
Interval  of  the 
Coefficient 

t 

p-value 

Math  score  in  2005-06 

0.52 

0.16 

0.21  0.84 

3.23 

0.00 

Reading  score  in  2005-06 

-0.33 

0.20 

-0.72  0.07 

-1.60 

0.12 

Female 

1.08 

0.92 

-0.73  2.88 

1.17 

0.25 

Minority 

0.19 

0.28 

-0.35  0.73 

0.70 

0.49 

IEP 

-2.22 

1.12 

-4.42  -0.03 

-1.98 

0.06 

Disadvantaged  students 

-1.14 

0.25 

-1.64  -0.64 

-4.50 

0.00 

Leader  participation,  2003-04  to  2006-07 

0.00 

0.00 

0.00  0.00 

1.03 

0.31 

Teacher  participation,  2003-04  to  2006-07 

0.00 

0.00 

-0.01  0.01 

-0.13 

0.90 
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Table  A.17 

Descriptive  Statistics  of  Variables  Included  in  District-Level  Regression  Analysis  for  Mathematics, 
Cohort  M  (Grade  5  in  2006-07) 


Variable 

N 

Mean 

Std.  Dev. 

Min. 

Max. 

Math  score  in  2006-07 

38 

-0.02 

0.41 

-0.90 

0.74 

Math  score  in  2004-05 

38 

-0.05 

0.38 

-0.91 

0.71 

Reading  score  in  2004-05 

38 

-0.07 

0.40 

-1.39 

0.56 

Female 

38 

0.52 

0.03 

0.44 

0.59 

Minority 

38 

0.21 

0.26 

0.00 

0.94 

IEP 

38 

0.15 

0.06 

0.06 

0.32 

Disadvantaged  students 

38 

0.43 

0.27 

0.05 

1.00 

Leader  participation,  2003-04  to  2006-07 

38 

1,422 

627 

92 

2,384 

Teacher  participation,  2003-04  to  2006-07 

38 

21.10 

12.48 

1.39 

48.60 

Table  A.18 

District-Level  Regression  Results  for  Mathematics, 

Cohort  M  (Grade  5  in  2006-1 

07) 

Variable 

Coefficient  Std.  Err. 

95%  Confidence 
Interval  of  the 
Coefficient 

t 

p-value 

Math  score  in  2004-05 

0.43 

0.20 

0.04  0.82 

2.16 

0.04 

Reading  score  in  2004-05 

0.08 

0.26 

-0.42  0.59 

0.32 

0.75 

Female 

-0.57 

1.13 

-2.77  1.64 

-0.50 

0.62 

Minority 

-0.16 

0.26 

-0.67  0.34 

-0.63 

0.53 

IEP 

-2.74 

0.76 

-4.24  -1.25 

-3.60 

0.00 

Disadvantaged  students 

-0.33 

0.30 

-0.92  0.26 

-1.09 

0.28 

Leader  participation,  2003-04  to  2006-07 

0.00 

0.00 

0.00  0.00 

1.05 

0.30 

Teacher  participation,  2003-04  to  2006-07 

0.00 

0.00 

-0.01  0.00 

-1.36 

0.18 
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Table  A.19 

Descriptive  Statistics  of  Variables  Included  in  District-Level  Regression  Analysis  for  Mathematics, 
Cohort  N  (Grade  4  in  2006-07) 


Variable 

N 

Mean 

Std.  Dev. 

Min. 

Max. 

Math  score  in  2006-07 

44 

-0.04 

0.39 

-1.00 

0.68 

Math  score  in  2005-06 

44 

-0.05 

0.35 

-0.80 

0.66 

Reading  score  in  2005-06 

44 

-0.05 

0.35 

-0.87 

0.76 

Female 

44 

0.50 

0.04 

0.39 

0.61 

Minority 

44 

0.19 

0.25 

0.01 

1.00 

IEP 

44 

0.15 

0.06 

0.00 

0.26 

Disadvantaged  students 

44 

0.40 

0.26 

0.05 

1.00 

Leader  participation,  2003-04  to  2006-07 

44 

1,425 

608 

92 

2,384 

Teacher  participation,  2003-04  to  2006-07 

44 

20.90 

12.78 

1.39 

48.60 

Table  A. 20 

District-Level  Regression  Results  for  Mathematics, 

Cohort  N  (Grade  4  in  2006-07) 

Variable 

Coefficient  Std.  Err. 

95%  Confidence 
Interval  of  the 
Coefficient 

t 

p-value 

Math  score  in  2005-06 

0.43 

0.25 

-0.06  0.92 

1.70 

0.10 

Reading  score  in  2005-06 

0.28 

0.26 

-0.24  0.80 

1.05 

0.30 

Female 

0.82 

0.85 

-0.85  2.48 

0.96 

0.34 

Minority 

-0.16 

0.25 

-0.65  0.33 

-0.64 

0.52 

IEP 

-0.46 

0.64 

-1.71  0.79 

-0.72 

0.48 

Disadvantaged  students 

-0.11 

0.25 

-0.59  0.37 

-0.44 

0.67 

Leader  participation,  2003-04  to  2006-07 

0.00 

0.00 

0.00  0.00 

1.96 

0.06 

Teacher  participation,  2003-04  to  2006-07 

0.00 

0.00 

-0.01  0.01 

-0.17 

0.87 
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District- Level  Analysis  Results  for  Science 

Tables  A.21  through  A.36  present  the  district-level  results  for  science  for  each  cohort.  All  test 
scores  are  standardized.  Female,  minority,  IEP,  and  disadvantaged  students  represent  the  pro¬ 
portions  of  those  students  in  a  district.  Leader  participation  is  the  sum  over  the  indicated  years 
of  district  leadership  participation;  teacher  participation  is  the  sum  over  the  indicated  years  of 
adjusted  district-level  science  teacher  participation.  These  measures  are  described  in  Chapter 
Four. 


Table  A.21 

Descriptive  Statistics  of  Variables  Included  in  District-Level  Regression  Analysis  for  Science,  Cohort  F 
(Grade  10  in  2004-05) 


Variable 

N 

Mean 

Std.  Dev. 

Min. 

Max. 

Science  score  in  2004-05 

22 

-0.07 

0.33 

-0.95 

0.48 

Math  score  in  2002-03 

22 

-0.02 

0.33 

-0.62 

0.56 

Reading  score  in  2002-03 

22 

-0.02 

0.35 

-0.76 

0.49 

Female 

22 

0.49 

0.04 

0.44 

0.59 

Minority 

22 

0.16 

0.21 

0.00 

0.68 

IEP 

22 

0.11 

0.09 

0.00 

0.33 

Disadvantaged  students 

22 

0.31 

0.26 

0.00 

1.00 

Leader  participation,  2003-04  to  2004-05 

22 

765 

234 

92 

1,262 

Teacher  participation,  2003-04  to  2004-05 

22 

5.61 

3.59 

0.80 

12.88 

Table  A. 22 

District-Level  Regression  Results  for  Science,  Cohort  F  (Grade  10  in  2004-05) 


Variable 

Coefficient 

Std.  Err. 

95%  Confidence 
Interval  of  the 
Coefficient 

t 

p-value 

Math  score  in  2002-03 

0.86 

0.43 

0.02 

1.70 

2.00 

0.07 

Reading  score  in  2002-03 

-0.27 

0.45 

-1.16 

0.61 

-0.61 

0.55 

Female 

-3.23 

1.87 

-6.89 

0.44 

-1.73 

0.11 

Minority 

-0.14 

0.33 

-0.78 

0.50 

-0.43 

0.68 

IEP 

0.05 

0.45 

-0.84 

0.94 

0.10 

0.92 

Disadvantaged  students 

-0.25 

0.21 

-0.66 

0.17 

-1.16 

0.27 

Leader  participation,  2003-04  to  2004-05 

0.00 

0.00 

0.00 

0.00 

-0.46 

0.65 

Teacher  participation,  2003-04  to  2004-05 

-0.01 

0.01 

-0.04 

0.01 

-1.14 

0.28 
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Table  A. 23 

Descriptive  Statistics  of  Variables  Included  in  District-Level  Regression  Analysis  for  Science,  Cohort  G 
(Grade  10  in  2005-06) 


Variable 

N 

Mean 

Std.  Dev. 

Min. 

Max. 

Science  score  in  2005-06 

40 

-0.09 

0.29 

-0.86 

0.41 

Math  score  in  2003-04 

40 

0.01 

0.38 

-0.81 

0.75 

Reading  score  in  2003-04 

40 

0.01 

0.35 

-0.80 

0.61 

Female 

40 

0.50 

0.04 

0.39 

0.61 

Minority 

40 

0.12 

0.14 

0.00 

0.67 

IEP 

40 

0.12 

0.16 

0.00 

1.00 

Disadvantaged  students 

40 

0.24 

0.17 

0.00 

0.71 

Leader  participation,  2003-04  to  2005-06 

40 

1,137 

413 

279 

1,886 

Teacher  participation,  2003-04  to  2005-06 

40 

16.38 

9.50 

3.17 

35.66 

Table  A. 24 

District-Level  Regression  Results  for  Science,  Cohort  G  (Grade  10  in  2005-06) 


Variable 

Coefficient 

Std.  Err. 

95%  Confidence 
Interval  of  the 
Coefficient 

t 

p-value 

Math  score  in  2003-04 

0.12 

0.13 

-0.14 

0.37 

0.91 

0.37 

Reading  score  in  2003-04 

0.50 

0.13 

0.23 

0.76 

3.68 

0.00 

Female 

-0.80 

0.54 

-1.87 

0.26 

-1.48 

0.15 

Minority 

0.15 

0.17 

-0.19 

0.48 

0.85 

0.40 

IEP 

0.15 

0.09 

-0.03 

0.34 

1.66 

0.11 

Disadvantaged  students 

-0.55 

0.16 

-0.86 

-0.24 

-3.53 

0.00 

Leader  participation,  2003-04  to  2005-06 

0.00 

0.00 

0.00 

0.00 

-1.04 

0.31 

Teacher  participation,  2003-04  to  2005-06 

0.00 

0.00 

0.00 

0.01 

0.53 

0.60 
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Table  A.25 

Descriptive  Statistics  of  Variables  Included  in  District-Level  Regression  Analysis  for  Science,  Cohort  H 
(Grade  10  in  2006-07) 


Variable 

N 

Mean 

Std.  Dev. 

Min. 

Max. 

Science  score  in  2006-07 

35 

-0.08 

0.33 

-0.90 

0.58 

Science  score  in  2003-04 

35 

0.02 

0.31 

-0.59 

0.50 

Female 

35 

0.49 

0.05 

0.39 

0.56 

Minority 

35 

0.16 

0.21 

0.00 

0.84 

IEP 

35 

0.11 

0.06 

0.00 

0.38 

Disadvantaged  students 

35 

0.32 

0.23 

0.03 

1.00 

Leader  participation,  2003-04  to  2006-07 

35 

1,496 

582 

340 

2,384 

Teacher  participation,  2003-04  to  2006-07 

35 

20.62 

11.35 

3.54 

44.05 

Table  A.26 

District-Level  Regression  Results  for  Science,  Cohort  H  (Grade  10  in  2006-07) 


Variable 

Coefficient 

Std.  Err. 

95%  Confidence 
Interval  of  the 
Coefficient 

t 

p-value 

Science  score  in  2003-04 

0.63 

0.15 

0.34 

0.92 

4.25 

0.00 

Female 

0.40 

0.58 

-0.74 

1.53 

0.68 

0.50 

Minority 

-0.39 

0.23 

-0.83 

0.05 

-1.72 

0.10 

IEP 

0.15 

0.55 

-0.94 

1.23 

0.26 

0.79 

Disadvantaged  students 

-0.35 

0.20 

-0.74 

0.03 

-1.79 

0.08 

Leader  participation,  2003-04  to  2006-07 

0.00 

0.00 

0.00 

0.00 

0.42 

0.68 

Teacher  participation,  2003-04  to  2006-07 

0.00 

0.00 

-0.01 

0.01 

0.19 

0.85 
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Table  A.27 

Descriptive  Statistics  of  Variables  Included  in  District-Level  Regression  Analysis  for  Science,  Cohort  I 
(Grade  7  in  2004-05) 


Variable 

N 

Mean 

Std.  Dev. 

Min. 

Max. 

Science  score  in  2004-05 

31 

-0.02 

0.33 

-0.74 

0.54 

Math  score  in  2002-03 

31 

0.02 

0.35 

-0.86 

0.62 

Reading  score  in  2002-03 

31 

0.04 

0.35 

-0.82 

0.66 

Female 

31 

0.48 

0.04 

0.39 

0.57 

Minority 

31 

0.17 

0.21 

0.00 

0.71 

IEP 

31 

0.09 

0.08 

0.00 

0.24 

Disadvantaged  students 

31 

0.35 

0.28 

0.00 

1.00 

Leader  participation,  2003-04  to  2004-05 

31 

685 

266 

92 

1,262 

Teacher  participation,  2003-04  to  2004-05 

31 

5.01 

3.35 

0.80 

12.88 

Table  A. 28 

District-Level  Regression  Results  for  Science,  Cohort  I  (Grade  7  in  2004-05) 


Variable 

Coefficient 

Std.  Err. 

95%  Confidence 
Interval  of  the 
Coefficient 

t 

p-value 

Math  score  in  2002-03 

0.10 

0.36 

-0.60 

0.79 

0.28 

0.79 

Reading  score  in  2002-03 

0.25 

0.34 

-0.43 

0.92 

0.71 

0.48 

Female 

0.73 

1.17 

-1.57 

3.03 

0.62 

0.54 

Minority 

-0.41 

0.32 

-1.03 

0.22 

-1.28 

0.21 

IEP 

0.01 

0.53 

-1.03 

1.06 

0.03 

0.98 

Disadvantaged  students 

-0.40 

0.22 

-0.83 

0.03 

-1.83 

0.08 

Leader  participation,  2003-04  to  2004-05 

0.00 

0.00 

0.00 

0.00 

0.48 

0.64 

Teacher  participation,  2003-04  to  2004-05 

0.01 

0.01 

-0.02 

0.04 

0.66 

0.51 
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Table  A. 29 

Descriptive  Statistics  of  Variables  Included  in  District-Level  Regression  Analysis  for  Science,  Cohort  J 
(Grade  7  in  2005-06) 


Variable 

N 

Mean 

Std.  Dev. 

Min. 

Max. 

Science  score  in  2005-06 

40 

-0.08 

0.36 

-0.89 

0.61 

Math  score  in  2003-04 

40 

0.08 

0.37 

-1.01 

0.76 

Reading  score  in  2003-04 

40 

0.08 

0.32 

-0.69 

0.68 

Female 

40 

0.49 

0.04 

0.42 

0.60 

Minority 

40 

0.16 

0.20 

0.00 

0.85 

IEP 

40 

0.12 

0.15 

0.00 

1.00 

Disadvantaged  students 

40 

0.34 

0.21 

0.00 

0.87 

Leader  participation,  2003-04  to  2005-06 

40 

1,137 

413 

279 

1,886 

Teacher  participation,  2003-04  to  2005-06 

40 

8.10 

4.94 

1.36 

19.16 

Table  A.30 

District-Level  Regression  Results  for  Science,  Cohort  J  (Grade  7  in  2005-06) 


Variable 

Coefficient 

Std.  Err. 

95%  Confidence 
Interval  of  the 
Coefficient 

t 

p-value 

Math  score  in  2003-04 

0.04 

0.21 

-0.37 

0.45 

0.19 

0.85 

Reading  score  in  2003-04 

0.36 

0.26 

-0.15 

0.88 

1.39 

0.17 

Female 

-0.14 

0.90 

-1.91 

1.63 

-0.16 

0.88 

Minority 

-0.50 

0.26 

-1.00 

0.01 

-1.91 

0.07 

IEP 

0.40 

0.21 

-0.01 

0.81 

1.91 

0.07 

Disadvantaged  students 

-0.69 

0.21 

-1.10 

-0.29 

-3.33 

0.00 

Leader  participation,  2003-04  to  2005-06 

0.00 

0.00 

0.00 

0.00 

-0.21 

0.84 

Teacher  participation,  2003-04  to  2005-06 

0.00 

0.01 

-0.02 

0.01 

-0.55 

0.59 
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Table  A.31 

Descriptive  Statistics  of  Variables  Included  in  District-Level  Regression  Analysis  for  Science,  Cohort  K 
(Grade  7  in  2006-07) 


Variable 

N 

Mean 

Std.  Dev. 

Min. 

Max. 

Science  score  in  2006-07 

35 

-0.11 

0.38 

-1.04 

0.51 

Science  score  in  2003-04 

35 

0.00 

0.32 

-0.72 

0.51 

Female 

35 

0.47 

0.04 

0.41 

0.56 

Minority 

35 

0.18 

0.22 

0.00 

0.89 

IEP 

35 

0.14 

0.06 

0.00 

0.30 

Disadvantaged  students 

35 

0.36 

0.24 

0.00 

1.00 

Leader  participation,  2003-04  to  2006-07 

35 

1,496 

582 

340 

2,384 

Teacher  participation,  2003-04  to  2006-07 

35 

10.11 

5.99 

1.61 

26.43 

Table  A.32 

District-Level  Regression  Results  for  Science,  Cohort  K  (Grade  7  in  2006-07) 


Variable 

Coefficient 

Std.  Err. 

95%  Confidence 
Interval  of  the 
Coefficient 

t 

p-value 

Science  score  in  2003-04 

0.89 

0.17 

0.56 

1.22 

5.35 

0.00 

Female 

-0.23 

0.70 

-1.61 

1.15 

-0.32 

0.75 

Minority 

-0.26 

0.23 

-0.71 

0.19 

-1.12 

0.27 

IEP 

-0.18 

0.56 

-1.28 

0.92 

-0.32 

0.75 

Disadvantaged  students 

-0.14 

0.21 

-0.56 

0.27 

-0.67 

0.51 

Leader  participation,  2003-04  to  2006-07 

0.00 

0.00 

0.00 

0.00 

-0.33 

0.75 

Teacher  participation,  2003-04  to  2006-07 

0.00 

0.01 

-0.01 

0.01 

-0.25 

0.81 
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Table  A.33 

Descriptive  Statistics  of  Variables  Included  in  District-Level  Regression  Analysis  for  Science, 
Cohort  M  (Grade  4  in  2005-06) 


Variable 

N 

Mean 

Std.  Dev. 

Min. 

Max. 

Science  score  in  2005-06 

44 

-0.02 

0.32 

-0.77 

0.52 

Math  score  in  2004-05 

40 

0.17 

0.26 

-0.51 

0.70 

Reading  score  in  2004-05 

40 

0.10 

0.23 

-0.45 

0.49 

Female 

44 

0.49 

0.04 

0.44 

0.60 

Minority 

44 

0.16 

0.19 

0.00 

0.84 

IEP 

44 

0.16 

0.20 

0.00 

1.00 

Disadvantaged  students 

44 

0.34 

0.21 

0.00 

0.85 

Leader  participation,  2003-04  to  2005-06 

53 

1,001 

480 

92 

1,886 

Teacher  participation,  2003-04  to  2005-06 

53 

6.85 

5.03 

0.76 

19.16 

Table  A.34 

District-Level  Regression  Results  for  Science,  Cohort  M  (Grade  4  in  2005-06) 


Variable 

Coefficient 

Std.  Err. 

95%  Confidence 
Interval  of  the 
Coefficient 

t 

p-value 

Math  score  in  2004-05 

0.15 

0.25 

-0.33 

0.63 

0.60 

0.55 

Reading  score  in  2004-05 

0.71 

0.30 

0.12 

1.31 

2.35 

0.03 

Female 

-0.05 

0.90 

-1.81 

1.71 

-0.06 

0.96 

Minority 

-0.83 

0.17 

-1.16 

-0.50 

-4.99 

0.00 

IEP 

0.15 

0.13 

-0.10 

0.40 

1.17 

0.25 

Disadvantaged  students 

0.03 

0.17 

-0.31 

0.37 

0.16 

0.87 

Leader  participation,  2003-04  to  2005-06 

0.00 

0.00 

0.00 

0.00 

-1.65 

0.11 

Teacher  participation,  2003-04  to  2005-06 

-0.01 

0.01 

-0.02 

0.01 

-0.78 

0.44 
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Table  A.35 

Descriptive  Statistics  of  Variables  Included  in  District-Level  Regression  Analysis  for  Science,  Cohort  N 
(Grade  4  in  2006-07) 


Variable 

N 

Mean 

Std.  Dev. 

Min. 

Max. 

Science  score  in  2006-07 

43 

-0.04 

0.34 

-0.80 

0.45 

Math  score  in  2005-06 

41 

0.15 

0.35 

-0.79 

0.84 

Reading  score  in  2005-06 

41 

-0.01 

0.30 

-0.62 

0.70 

Female 

43 

0.50 

0.04 

0.39 

0.62 

Minority 

43 

0.16 

0.20 

0.02 

0.80 

IEP 

43 

0.14 

0.07 

0.00 

0.30 

Disadvantaged  students 

43 

0.36 

0.24 

0.00 

1.00 

Leader  participation,  2003-04  to  2006-07 

53 

1,299 

633 

92 

2,384 

Teacher  participation,  2003-04  to  2006-07 

53 

8.74 

6.05 

0.80 

26.43 

Table  A.36 

District-Level  Regression  Results  for  Science,  Cohort  N  (Grade  4  in  2006-07) 


Variable 

Coefficient 

Std.  Err. 

95%  Confidence 
Interval  of  the 
Coefficient 

t 

p-value 

Math  score  in  2005-06 

0.06 

0.19 

-0.32 

0.44 

0.32 

0.75 

Reading  score  in  2005-06 

0.54 

0.23 

0.08 

0.99 

2.32 

0.03 

Female 

-0.27 

0.64 

-1.52 

0.97 

-0.43 

0.67 

Minority 

-0.83 

0.19 

-1.20 

-0.46 

-4.42 

0.00 

IEP 

0.15 

0.48 

-0.79 

1.10 

0.32 

0.75 

Disadvantaged  students 

0.03 

0.19 

-0.34 

0.40 

0.16 

0.88 

Leader  participation,  2003-04  to  2006-07 

0.00 

0.00 

0.00 

0.00 

0.33 

0.75 

Teacher  participation,  2003-04  to  2006-07 

0.00 

0.01 

-0.01 

0.01 

0.04 

0.97 
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State-Level  Analysis  Results  for  Mathematics 


Tables  A.37  through  A.39  show  the  balance  in  pretreatment  covariates  before  and  after  match¬ 
ing  for  the  three  comparison  groups.  Tables  A.40  and  A.41  present  the  state-level  regression 
results  for  mathematics  for  two  cohorts. 

Table  A.37 

Balance  in  Covariates,  Before  and  After  Matching,  Comparison  Group  1 

Covariate 

MSP  Group 
Mean 

Comparison  Group  Mean 

Standardized  Bias 

Unweighted 

Weighted 

Unweighted 

Weighted 

PSSA  test  score  means 

1998  grade  11  reading 

1,281 

1,299 

1,288 

-0.28 

-0.12 

1998  grade  11  math 

1,266 

1,295 

1,273 

-0.32 

-0.08 

1998  grade  8  reading 

1,291 

1,314 

1,298 

-0.29 

-0.09 

1998  grade  8  math 

1,289 

1,311 

1,295 

-0.25 

-0.06 

1998  grade  5  reading 

1,310 

1,330 

1,319 

-0.21 

-0.10 

1998  grade  5  math 

1,302 

1,320 

1,311 

-0.24 

-0.12 

1999  grade  11  reading 

1,285 

1,301 

1,288 

-0.20 

-0.03 

1999  grade  11  math 

1,273 

1,302 

1,286 

-0.34 

-0.15 

1999  grade  8  reading 

1,290 

1,320 

1,305 

-0.39 

-0.19 

1999  grade  8  math 

1,287 

1,315 

1,299 

-0.30 

-0.13 

1999  grade  5  reading 

1,317 

1,333 

1,324 

-0.20 

-0.09 

1999  grade  5  math 

1,302 

1,319 

1,312 

-0.21 

-0.12 

2000  grade  11  reading 

1,275 

1,302 

1,289 

-0.39 

-0.20 

2000  grade  11  math 

1,282 

1,307 

1,291 

-0.31 

-0.11 

2000  grade  8  reading 

1,305 

1,320 

1,309 

-0.20 

-0.06 

2000  grade  8  math 

1,299 

1,324 

1,310 

-0.27 

-0.12 

2000  grade  5  reading 

1,314 

1,332 

1,325 

-0.24 

-0.15 

2000  grade  5  math 

1,303 

1,325 

1,318 

-0.25 

-0.18 

2001  grade  11  reading 

1,291 

1,306 

1,299 

-0.21 

-0.11 

2001  grade  11  math 

1,291 

1,308 

1,294 

-0.21 

-0.04 

2001  grade  8  reading 

1,309 

1,323 

1,311 

-0.18 

-0.02 

2001  grade  8  math 

1,302 

1,325 

1,312 

-0.27 

-0.12 

2001  grade  5  reading 

1,316 

1,332 

1,327 

-0.22 

-0.15 

2001  grade  5  math 

1,313 

1,328 

1,322 

-0.19 

-0.11 

2002  grade  11  reading 

1,301 

1,328 

1,316 

-0.34 

-0.19 

2002  grade  11  math 

1,295 

1,322 

1,311 

-0.30 

-0.17 
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Table  A. 37 — Continued 


Comparison  Group  Mean  Standardized  Bias 


Covariate 

Mean 

Unweighted 

Weighted 

Unweighted 

Weighted 

2002  grade  8  reading 

1,315 

1,327 

1,315 

-0.15 

-0.01 

2002  grade  8  math 

1,311 

1,330 

1,326 

-0.22 

-0.17 

2002  grade  5  reading 

1,325 

1,339 

1,334 

-0.17 

-0.12 

2002  grade  5  math 

1,319 

1,333 

1,330 

-0.16 

-0.14 

2003  grade  11  reading 

1,321 

1,334 

1,322 

-0.16 

-0.01 

2003  grade  11  math 

1,314 

1,327 

1,315 

-0.16 

-0.02 

2003  grade  8  reading 

1,337 

1,358 

1,344 

-0.24 

-0.07 

2003  grade  8  math 

1,316 

1,336 

1,325 

-0.22 

-0.10 

2003  grade  5  reading 

1,346 

1,358 

1,356 

-0.13 

-0.12 

2003  grade  5  math 

1,348 

1,360 

1,362 

-0.14 

-0.17 

2004  grade  11  reading 

1,342 

1,363 

1,352 

-0.19 

-0.09 

2004  grade  11  math 

1,321 

1,333 

1,321 

-0.13 

0.00 

2004  grade  8  reading 

1,362 

1,386 

1,369 

-0.25 

-0.07 

2004  grade  8  math 

1,336 

1,367 

1,356 

-0.33 

-0.22 

2004  grade  5  reading 

1,386 

1,395 

1,388 

-0.09 

-0.02 

2004  grade  5  math 

1,395 

1,404 

1,398 

-0.09 

-0.02 

Median  absolute  standardized  bias- 

—PSSA  test  score 

means 

0.22 

0.11 

Time  trend  in  PSSA  score,  slope 

Grade  11  reading 

7.64 

7.33 

7.52 

0.03 

0.01 

Grade  11  math 

8.94 

6.28 

8.21 

0.28 

0.08 

Grade  8  reading 

8.89 

6.98 

7.48 

0.22 

0.16 

Grade  8  math 

6.01 

4.87 

6.75 

0.15 

-0.10 

Grade  5  reading 

5.77 

4.47 

6.24 

0.12 

-0.04 

Grade  5  math 

8.40 

6.94 

8.97 

0.14 

-0.06 

Median  absolute  standardized  bias- 

—time  trend  in  PSSA  score,  slope 

0.15 

0.07 

2001  racial  distribution  of  students 

Percentage  of  black  students 

0.18 

0.04 

0.09 

0.56 

0.36 

Percentage  of  Hispanic  students 

0.00 

0.02 

0.03 

-8.15 

-14.163 

Percentage  of  American  Indian 
students 

0.00 

0.00 

0.00 

-0.11 

-0.17 

Percentage  of  Asian  students 

0.01 

0.01 

0.01 

-0.30 

-0.30 

Percentage  of  white  students 

0.81 

0.93 

0.86 

-0.49 

-0.23 
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Table  A. 37 — Continued 


MSP  Group 
Mean 

Comparison  Group  Mean 

Standardized  Bias 

Covariate 

Unweighted 

Weighted 

Unweighted 

Weighted 

Transformed  percentage  of  white 
students 

0.34 

-0.10 

-0.20 

-0.35 

-0.20 

Percentage  low  income 

37.28 

25.67 

31.80 

0.53 

0.25 

Overall  attendance  rate 

93.52 

94.55 

94.06 

-0.66 

-0.35 

Overall  attendance  rate  information  is 
missing 

0.06 

0.08 

0.09 

-0.05 

-0.12 

Graduation  total  rate 

90.08 

90.65 

89.13 

-0.08 

-0.08 

Median  absolute  standardized  bias — 2001  racial  distribution  of  students 

District  population,  2000  census 

0.42 

0.24 

Percentage  black  (population) 

0.09 

0.03 

0.05 

0.50 

0.32 

Percentage  white  (population) 

0.89 

0.95 

0.92 

-0.45 

-0.23 

Percentage  other  race  (population) 

0.02 

0.03 

0.03 

-0.27 

-0.87 

Percentage  urban 

0.78 

0.57 

0.73 

0.69 

0.16 

Percentage  unmarried  heads  of 
household  with  children 

0.29 

0.19 

0.24 

0.63 

0.30 

Percentage  of  female  heads  of 
household  with  children 

0.23 

0.14 

0.19 

0.66 

0.32 

Percentage  of  men  with  0-8  years  of 
education 

0.05 

0.06 

0.05 

-0.42 

-0.27 

Percentage  of  men  with  9-11  years  of 
education 

0.08 

0.10 

0.10 

-0.44 

-0.44 

Percentage  of  men  with  12  years  of 
education 

0.42 

0.44 

0.44 

-0.19 

-0.11 

Percentage  of  men  with  13-15  years  of 
education 

0.23 

0.20 

0.21 

0.66 

0.46 

Percentage  of  men  with  16+  years  of 
education 

0.22 

0.20 

0.21 

0.11 

0.09 

Percentage  of  women  with  0-8  years 
of  education 

0.06 

0.06 

0.06 

-0.09 

-0.17 

Percentage  of  women  with  9-11  years 
of  education 

0.09 

0.09 

0.10 

-0.22 

-0.41 

Percentage  of  women  with  12  years  of 
education 

0.45 

0.46 

0.46 

-0.20 

-0.12 

Percentage  of  women  with  13-15  years 
of  education 

0.23 

0.21 

0.21 

0.60 

0.53 

Percentage  of  women  with  16+  years 
of  education 

0.18 

0.18 

0.18 

0.00 

0.05 

Percentage  with  13+  years  of  education 

0.43 

0.40 

0.40 

0.25 

0.22 

Percentage  with  16+  years  of  education 

0.20 

0.19 

0.19 

0.06 

0.07 
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Table  A. 37 — Continued 


MSP  Group 
Mean 

Comparison  Group  Mean 

Standardized  Bias 

Covariate 

Unweighted 

Weighted 

Unweighted 

Weighted 

Percentage  with  master's  degree 

0.05 

0.05 

0.05 

-0.01 

0.02 

Percentage  of  men  not  in  the  labor 
force 

0.34 

0.30 

0.32 

0.55 

0.19 

Percentage  of  women  not  in  the  labor 
force 

0.48 

0.45 

0.47 

0.62 

0.29 

Percentage  of  population  age  16+  not 
in  the  labor  force 

0.41 

0.38 

0.40 

0.64 

0.26 

Percentage  of  population  not  working 
in  Pennsylvania 

0.03 

0.05 

0.04 

-0.33 

-0.11 

Percentage  under  18 

0.23 

0.24 

0.23 

-0.49 

-0.18 

Percentage  over  65 

0.19 

0.16 

0.18 

0.69 

0.24 

Median  rent  ($) 

471 

509 

497 

-0.38 

-0.26 

Median  real  estate  tax  ($) 

1,516 

1,537 

1,526 

-0.04 

-0.02 

Median  home  value  ($) 

77,270 

98,410 

90,730 

-0.73 

-0.46 

Percentage  of  vacant  houses 

0.08 

0.10 

0.09 

-0.57 

-0.43 

Percentage  of  owner-occupied  houses 

0.71 

0.77 

0.72 

-0.52 

-0.15 

1999  median  household  income  ($) 

35,810 

41,570 

39,020 

-0.58 

-0.32 

1999  median  family  income  ($) 

44,680 

49,420 

47,350 

-0.41 

-0.23 

1999  per  capita  income  ($) 

19,410 

20,040 

19,680 

-0.11 

-0.05 

Median  absolute  standardized  bias — district  population,  2000  census 

0.44 

0.23 

Population 

Population  per  square  mile 

1,844 

887 

1,585 

0.49 

0.13 

Transformed  population  per  square 
mile 

6.83 

5.67 

6.42 

0.87 

0.31 

Total  district  population 

18,280 

21,170 

25,200 

-0.26 

-0.62 

Median  absolute  standardized  bias — population 

0.49 

0.31 

2002  district  financial  data 

Average  daily  membership  (ADM) 

2,555 

3,187 

3,605 

-0.42 

-0.69 

Transformed  ADM 

2.04 

2.05 

2.06 

-0.16 

-0.36 

Percentage  of  students  in  families 
receiving  welfare 

0.06 

0.02 

0.04 

0.60 

0.31 

Pupil-teacher  ratio 

15.20 

15.78 

15.71 

-0.36 

-0.32 

Personal  income  tax  effort 

44.30 

44.73 

45.88 

-0.04 

-0.15 

Transformed  personal  income  tax 
effort 

3.77 

3.76 

3.79 

0.01 

-0.1 

Average  teacher  salary  ($) 

49,570 

49,810 

50,570 

-0.05 

-0.21 
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Table  A. 37 — Continued 


MSP  Group 
Mean 

Comparison  Group  Mean 

Standardized  Bias 

Covariate 

Unweighted 

Weighted 

Unweighted 

Weighted 

Transformed  average  teacher  salary 

10.81 

10.81 

10.82 

-0.02 

-0.19 

Ratio  of  district  market  value  to  total 
personal  income 

0.59 

0.55 

0.56 

0.24 

0.20 

Ratio  of  market  value  to  personal 
income  tax  effort 

24.18 

20.17 

21.75 

0.65 

0.39 

Transformed  ratio  of  market  value  to 
income  tax  effort 

4.88 

4.47 

4.64 

0.69 

0.41 

Ratio  of  instructional  expense  to 

ADM  ($) 

5,888 

5,708 

5,795 

0.20 

0.10 

Transformed  ratio  of  instructional 
expense  to  ADM 

8.67 

8.64 

8.65 

0.23 

0.12 

Teacher  average  years  of  service 

15.97 

16.69 

16.80 

-0.33 

-0.38 

Median  absolute  standardized  bias — 2002  district  financial  data 

0.24 

0.26 

Interactions 

Interaction:  ADM,  ratio  of  market  value 
to  income 

1,421 

1,589 

1,930 

-0.21 

-0.63 

Interaction:  ADM,  tax  effort 

60,550 

67,320 

82,210 

-0.19 

-0.59 

Interaction:  ADM,  pupil-teacher  ratio 

39,520 

51,410 

57,670 

-0.49 

-0.74 

Interaction:  ADM,  population  per 
square  mile 

4,387,000 

4,510,000 

8,421,000 

-0.02 

-0.76 

Median  absolute  standardized  bias — interactions 

0.20 

0.69 

Median  absolute  standardized  bias — all  covariates 

0.25 

0.16 

NOTE:  Comparison  group  1  (n  =  442  school  districts)  consists  of  non-MSP  Pennsylvania  school  districts,  excluding 
Pittsburgh  and  Philadelphia  city  school  districts  and  a  district  with  no  enrolled  students.  The  MSP  group  (n  =  48 
school  districts)  excludes  six  MSP  districts  that  did  not  join  the  project  until  2006. 


a  The  standardized  bias  of  this  covariate  is  inflated  because  there  is  little  variation  within  the  MSP  group;  using 
the  standard  deviation  across  all  districts  produces  a  weighted  standardized  bias  of  -0.56. 
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Table  A.38 

Balance  in  Covariates,  Before  and  After  Matching,  Comparison  Group  2 


Comparison  Group  Mean  Standardized  Bias 

MSP  Group - 

Covariate  Mean  Unweighted  Weighted  Unweighted  Weighted 

PSSA  test  score  means 


1998  grade  11  reading 

1,281 

1,299 

1,287 

-0.28 

-0.10 

1998  grade  11  math 

1,266 

1,295 

1,272 

-0.31 

-0.07 

1998  grade  8  reading 

1,291 

1,313 

1,296 

-0.28 

-0.06 

1998  grade  8  math 

1,289 

1,310 

1,292 

-0.24 

-0.04 

1998  grade  5  reading 

1,310 

1,329 

1,317 

-0.20 

-0.07 

1998  grade  5  math 

1,302 

1,318 

1,308 

-0.22 

-0.08 

1999  grade  11  reading 

1,285 

1,300 

1,284 

-0.19 

0.01 

1999  grade  11  math 

1,273 

1,301 

1,283 

-0.33 

-0.12 

1999  grade  8  reading 

1,290 

1,319 

1,302 

-0.38 

-0.15 

1999  grade  8  math 

1,287 

1,313 

1,296 

-0.29 

-0.10 

1999  grade  5  reading 

1,317 

1,332 

1,322 

-0.20 

-0.07 

1999  grade  5  math 

1,302 

1,317 

1,309 

-0.18 

-0.08 

2000  grade  11  reading 

1,275 

1,301 

1,287 

-0.39 

-0.18 

2000  grade  11  math 

1,282 

1,306 

1,288 

-0.30 

-0.08 

2000  grade  8  reading 

1,305 

1,319 

1,307 

-0.19 

-0.03 

2000  grade  8  math 

1,299 

1,323 

1,307 

-0.25 

-0.08 

2000  grade  5  reading 

1,314 

1,331 

1,323 

-0.23 

-0.11 

2000  grade  5  math 

1,303 

1,324 

1,315 

-0.23 

-0.14 

2001  grade  11  reading 

1,291 

1,306 

1,298 

-0.20 

-0.09 

2001  grade  11  math 

1,291 

1,308 

1,292 

-0.20 

-0.02 

2001  grade  8  reading 

1,309 

1,322 

1,308 

-0.17 

0.02 

2001  grade  8  math 

1,302 

1,324 

1,309 

-0.25 

-0.08 

2001  grade  5  reading 

1,316 

1,331 

1,324 

-0.20 

-0.10 

2001  grade  5  math 

1,313 

1,326 

1,319 

-0.16 

-0.07 

2002  grade  11  reading 

1,301 

1,327 

1,314 

-0.34 

-0.17 

2002  grade  11  math 

1,295 

1,322 

1,309 

-0.30 

-0.16 

2002  grade  8  reading 

1,315 

1,326 

1,313 

-0.14 

0.02 

2002  grade  8  math 

1,311 

1,329 

1,323 

-0.20 

-0.14 

2002  grade  5  reading 

1,325 

1,337 

1,332 

-0.15 

-0.08 

2002  grade  5  math 

1,319 

1,331 

1,327 

-0.14 

-0.10 

2003  grade  11  reading 

1,321 

1,334 

1,321 

-0.16 

0.01 
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Table  A. 38 — Continued 


MSP  Group 
Mean 

Comparison  Group  Mean 

Standardized  Bias 

Covariate 

Unweighted 

Weighted 

Unweighted 

Weighted 

2003  grade  11  math 

1,314 

1,327 

1,313 

-0.16 

0.01 

2003  grade  8  reading 

1,337 

1,357 

1,342 

-0.23 

-0.06 

2003  grade  8  math 

1,316 

1,335 

1,324 

-0.21 

-0.09 

2003  grade  5  reading 

1,346 

1,356 

1,355 

-0.12 

-0.10 

2003  grade  5  math 

1,348 

1,358 

1,360 

-0.12 

-0.14 

2004  grade  11  reading 

1,342 

1,364 

1,351 

-0.20 

-0.08 

2004  grade  11  math 

1,321 

1,334 

1,320 

-0.13 

0.01 

2004  grade  8  reading 

1,362 

1,385 

1,367 

-0.25 

-0.05 

2004  grade  8  math 

1,336 

1,366 

1,355 

-0.32 

-0.20 

2004  grade  5  reading 

1,386 

1,394 

1,386 

-0.08 

0.00 

2004  grade  5  math 

1,395 

1,403 

1,396 

-0.07 

0.00 

Median  absolute  standardized  bias— 

-PSSA  test  score 

means 

0.20 

0.08 

Time  trend  in  PSSA  score,  slope 

Grade  11  reading 

7.64 

7.44 

7.63 

0.02 

0.00 

Grade  11  math 

8.94 

6.38 

8.18 

0.27 

0.08 

Grade  8  reading 

8.89 

6.97 

7.63 

0.22 

0.14 

Grade  8  math 

6.01 

4.91 

6.96 

0.14 

-0.12 

Grade  5  reading 

5.77 

4.34 

6.29 

0.13 

-0.05 

Grade  5  math 

8.40 

6.92 

9.08 

0.14 

-0.07 

Median  absolute  standardized  bias— 

-time  trend  in  PSSA  score,  slope 

0.14 

0.08 

2001  racial  distribution  of  students 

Percentage  of  black  students 

0.18 

0.04 

0.10 

0.58 

0.35 

Percentage  of  Hispanic  students 

0.00 

0.02 

0.03 

-6.63 

-12.62a 

Percentage  of  American  Indian 
students 

0.00 

0.00 

0.00 

-0.09 

-0.17 

Percentage  of  Asian  students 

0.01 

0.01 

0.01 

-0.17 

-0.23 

Percentage  of  white  students 

0.81 

0.94 

0.86 

-0.53 

-0.24 

Transformed  percentage  of  white 
students 

0.34 

-0.08 

-0.20 

-0.37 

-0.20 

Percentage  low  income 

37.28 

26.07 

32.69 

0.51 

0.21 

Overall  attendance  rate 

93.52 

94.56 

94.05 

-0.67 

-0.34 

Overall  attendance  rate  information 
missing 

is  0.06 

0.08 

0.11 

-0.07 

-0.19 

Graduation  total  rate 

90.08 

90.78 

89.05 

-0.10 

0.15 
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Table  A. 38 — Continued 


Comparison  Group  Mean  Standardized  Bias 

MSP  Group - 

Covariate  Mean  Unweighted  Weighted  Unweighted  Weighted 


Median  absolute  standardized  bias — 2001  racial  distribution  of  students  0.44  0.22 

District  population,  2000  census 


Percentage  black  (population) 

0.09 

0.02 

0.05 

0.52 

0.31 

Percentage  white  (population) 

0.89 

0.96 

0.92 

-0.49 

-0.23 

Percentage  other  race  (population) 

0.02 

0.02 

0.03 

-0.08 

-0.75 

Percentage  urban 

0.78 

0.54 

0.72 

0.76 

0.18 

Percentage  unmarried  heads  of 
household  with  children 

0.29 

0.19 

0.25 

0.64 

0.27 

Percentage  of  female  heads  of 
household  with  children 

0.23 

0.14 

0.19 

0.66 

0.30 

Percentage  of  men  with  0-8  years  of 
education 

0.05 

0.06 

0.06 

-0.48 

-0.29 

Percentage  of  men  with  9-11  years  of 
education 

0.08 

0.10 

0.10 

-0.49 

-0.51 

Percentage  of  men  with  12  years  of 
education 

0.42 

0.45 

0.44 

-0.26 

-0.16 

Percentage  of  men  with  13-15  years  of 
education 

0.23 

0.20 

0.21 

0.71 

0.50 

Percentage  of  men  with  16+  years  of 
education 

0.22 

0.20 

0.20 

0.17 

0.14 

Percentage  of  women  with  0-8  years 
of  education 

0.06 

0.06 

0.06 

-0.13 

-0.20 

Percentage  of  women  with  9-11  years 
of  education 

0.09 

0.09 

0.10 

-0.25 

-0.45 

Percentage  of  women  with  12  years  of 
education 

0.45 

0.47 

0.46 

-0.29 

-0.19 

Percentage  of  women  with  13-15  years 
of  education 

0.23 

0.21 

0.21 

0.66 

0.57 

Percentage  of  women  with  16+  years 
of  education 

0.18 

0.17 

0.17 

0.06 

0.10 

Percentage  with  13+  years  of  education 

0.43 

0.39 

0.39 

0.33 

0.28 

Percentage  with  16+  years  of  education 

0.20 

0.18 

0.18 

0.12 

0.12 

Percentage  with  master's  degree 

0.05 

0.04 

0.04 

0.05 

0.07 

Percentage  of  men  not  in  the  labor 
force 

0.34 

0.30 

0.33 

0.51 

0.15 

Percentage  of  women  not  in  the  labor 
force 

0.48 

0.46 

0.47 

0.55 

0.25 

Percentage  of  population  age  16+  not 
in  the  labor  force 

0.41 

0.38 

0.40 

0.58 

0.21 
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Table  A. 38 — Continued 


MSP  Group 
Mean 

Comparison  Group  Mean 

Standardized  Bias 

Covariate 

Unweighted 

Weighted 

Unweighted 

Weighted 

Percentage  of  population  not  working 
in  Pennsylvania 

0.03 

0.05 

0.04 

-0.28 

-0.07 

Percentage  under  18 

0.23 

0.24 

0.23 

-0.48 

-0.20 

Percentage  over  65 

0.19 

0.16 

0.18 

0.67 

0.25 

Median  rent  ($) 

471 

496 

488 

-0.26 

-0.17 

Median  real  estate  tax  ($) 

1,516 

1,460 

1,467 

0.10 

0.08 

Median  home  value  ($) 

77,270 

95,040 

88,160 

-0.61 

-0.37 

Percentage  of  vacant  houses 

0.08 

0.10 

0.10 

-0.69 

-0.51 

Percentage  of  owner-occupied  houses 

0.71 

0.77 

0.72 

-0.55 

-0.12 

1999  median  household  income  ($) 

35,810 

40,740 

38,270 

-0.50 

-0.25 

1999  median  family  income  ($) 

44,680 

48,400 

46,480 

-0.32 

-0.16 

1999  per  capita  income  ($) 

19,410 

19,640 

19,380 

-0.04 

0.01 

Median  absolute  standardized  bias — district  population,  2000  census 

0.48 

0.21 

Population 

Population  per  square  mile 

1,844 

754 

1,545 

0.56 

0.15 

Transformed  population  per  square 
mile 

6.83 

5.55 

6.38 

0.96 

0.34 

Total  district  population 

18,280 

19,620 

24,030 

-0.12 

-0.51 

Median  absolute  standardized  bias — population 

0.56 

0.34 

2002  district  financial  data 

Average  daily  membership  (ADM) 

2,555 

2,979 

3,466 

-0.28 

-0.60 

Transformed  ADM 

2.04 

2.04 

2.06 

-0.07 

-0.31 

Percentage  of  students  in  families 
receiving  welfare 

0.06 

0.02 

0.04 

0.61 

0.29 

Pupil-teacher  ratio 

15.20 

15.78 

15.74 

-0.36 

-0.34 

Personal  income  tax  effort 

44.30 

44.05 

45.65 

0.02 

-0.13 

Transformed  personal  income  tax 
effort 

3.77 

3.75 

3.78 

0.07 

-0.08 

Average  teacher  salary  ($) 

49,570 

49,170 

50,200 

0.08 

-0.13 

Transformed  average  teacher  salary 

10.81 

10.80 

10.82 

0.11 

-0.11 

Ratio  of  district  market  value  to  total 
personal  income 

0.59 

0.56 

0.57 

0.18 

0.13 

Ratio  of  market  value  to  personal 
income  tax  effort 

24.18 

19.95 

21.78 

0.69 

0.39 

Transformed  ratio  of  market  value  to 

4.88 

4.44 

4.64 

0.73 

0.40 

income  tax  effort 
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Table  A. 38 — Continued 


MSP  Group 
Mean 

Comparison  Group  Mean 

Standardized  Bias 

Covariate 

Unweighted 

Weighted 

Unweighted 

Weighted 

Ratio  of  instructional  expense  to 
ADM  ($) 

5,888 

5,618 

5,736 

0.30 

0.17 

Transformed  ratio  of  instructional 
expense  to  ADM 

8.67 

8.62 

8.64 

0.32 

0.18 

Teacher  average  years  of  service 

15.97 

16.79 

16.88 

-0.38 

-0.42 

Median  absolute  standardized  bias — 

2002  district  financial  data 

0.29 

0.24 

Interactions 

Interaction:  ADM,  ratio  of  market  value  1,421 

to  income 

1,518 

1,901 

-0.12 

-0.59 

Interaction:  ADM,  tax  effort 

60,550 

61,840 

79,400 

-0.04 

-0.52 

Interaction:  ADM,  pupil-teacher  ratio 

39,520 

48,030 

55,360 

-0.35 

-0.65 

Interaction:  ADM,  population  per 
square  mile 

4,387,000 

3,436,000 

7,748,000 

0.18 

-0.63 

Median  absolute  standardized  bias — 

interactions 

0.15 

0.61 

Median  absolute  standardized  bias — all  covariates 

0.25 

0.15 

NOTE:  Comparison  group  2  (n  =  412  school  districts)  consists  of  non-MSP  Pennsylvania  school  districts,  excluding 
Pittsburgh  and  Philadelphia  city  school  districts,  a  district  with  no  enrolled  students,  and  districts  that 
participated  in  the  Philadelphia  math  and  science  partnership.  The  MSP  group  (n  =  48  school  districts)  excludes  six 
MSP  districts  that  did  not  join  the  project  until  2006. 

a  The  standardized  bias  of  this  covariate  is  inflated  because  there  is  little  variation  within  the  MSP  group;  using 
the  standard  deviation  across  all  districts  produces  a  weighted  standardized  bias  of  -0.50. 
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Table  A. 39 

Balance  in  Covariates,  Before  and  After  Matching,  Comparison  Group  3 


Comparison  Group  Mean  Standardized  Bias 

MSP  Group - 

Covariate  Mean  Unweighted  Weighted  Unweighted  Weighted 

PSSA  test  score  means 


1998  grade  11  reading 

1,281 

1,298 

1,283 

-0.26 

-0.04 

1998  grade  11  math 

1,266 

1,296 

1,268 

-0.32 

-0.03 

1998  grade  8  reading 

1,291 

1,312 

1,289 

-0.26 

0.02 

1998  grade  8  math 

1,289 

1,309 

1,287 

-0.23 

0.02 

1998  grade  5  reading 

1,310 

1,328 

1,315 

-0.20 

-0.05 

1998  grade  5  math 

1,302 

1,318 

1,307 

-0.21 

-0.07 

1999  grade  11  reading 

1,285 

1,300 

1,282 

-0.18 

0.04 

1999  grade  11  math 

1,273 

1,302 

1,282 

-0.35 

-0.10 

1999  grade  8  reading 

1,290 

1,319 

1,298 

-0.37 

-0.10 

1999  grade  8  math 

1,287 

1,312 

1,292 

-0.28 

-0.05 

1999  grade  5  reading 

1,317 

1,331 

1,320 

-0.18 

-0.04 

1999  grade  5  math 

1,302 

1,315 

1,307 

-0.17 

-0.07 

2000  grade  11  reading 

1,275 

1,303 

1,284 

-0.41 

-0.13 

2000  grade  11  math 

1,282 

1,307 

1,286 

-0.32 

-0.05 

2000  grade  8  reading 

1,305 

1,317 

1,303 

-0.16 

0.02 

2000  grade  8  math 

1,299 

1,323 

1,306 

-0.25 

-0.08 

2000  grade  5  reading 

1,314 

1,329 

1,318 

-0.20 

-0.05 

2000  grade  5  math 

1,303 

1,321 

1,311 

-0.21 

-0.10 

2001  grade  11  reading 

1,291 

1,305 

1,292 

-0.20 

-0.01 

2001  grade  11  math 

1,291 

1,308 

1,287 

-0.20 

0.04 

2001  grade  8  reading 

1,309 

1,322 

1,303 

-0.16 

0.08 

2001  grade  8  math 

1,302 

1,324 

1,307 

-0.26 

-0.06 

2001  grade  5  reading 

1,316 

1,328 

1,320 

-0.17 

-0.05 

2001  grade  5  math 

1,313 

1,324 

1,318 

-0.14 

-0.06 

2002  grade  11  reading 

1,301 

1,329 

1,314 

-0.36 

-0.17 

2002  grade  11  math 

1,295 

1,323 

1,310 

-0.31 

-0.16 

2002  grade  8  reading 

1,315 

1,325 

1,309 

-0.13 

0.06 

2002  grade  8  math 

1,311 

1,329 

1,325 

-0.21 

-0.16 

2002  grade  5  reading 

1,325 

1,336 

1,332 

-0.13 

-0.08 

2002  grade  5  math 

1,319 

1,329 

1,329 

-0.12 

-0.12 

2003  grade  11  reading 

1,321 

1,333 

1,314 

-0.14 

0.09 
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Table  A. 39 — Continued 


Comparison  Group  Mean  Standardized  Bias 


Covariate 

Mean 

Unweighted 

Weighted 

Unweighted 

Weighted 

2003  grade  11  math 

1,314 

1,326 

1,309 

-0.15 

0.05 

2003  grade  8  reading 

1,337 

1,357 

1,338 

-0.23 

-0.01 

2003  grade  8  math 

1,316 

1,335 

1,321 

-0.22 

-0.06 

2003  grade  5  reading 

1,346 

1,356 

1,353 

-0.11 

-0.09 

2003  grade  5  math 

1,348 

1,358 

1,360 

-0.12 

-0.14 

2004  grade  11  reading 

1,342 

1,363 

1,346 

-0.20 

-0.03 

2004  grade  11  math 

1,321 

1,333 

1,314 

-0.12 

0.06 

2004  grade  8  reading 

1,362 

1,385 

1,364 

-0.24 

-0.02 

2004  grade  8  math 

1,336 

1,366 

1,352 

-0.32 

-0.17 

2004  grade  5  reading 

1,386 

1,393 

1,379 

-0.07 

0.07 

2004  grade  5  math 

1,395 

1,402 

1,391 

-0.06 

0.05 

Median  absolute  standardized  bias— 

-PSSA  test  score  means 

0.20 

0.06 

Time  trend  in  PSSA  score,  slope 

Grade  11  reading 

7.64 

7.47 

7.45 

0.02 

0.02 

Grade  11  math 

8.94 

6.10 

8.25 

0.30 

0.07 

Grade  8  reading 

8.89 

7.14 

7.93 

0.20 

0.11 

Grade  8  math 

6.01 

5.19 

7.73 

0.11 

-0.22 

Grade  5  reading 

5.77 

4.26 

6.52 

0.14 

-0.07 

Grade  5  math 

8.40 

7.05 

9.49 

0.13 

-0.11 

Median  absolute  standardized  bias— 

-time  trend  in  PSSA  score,  slope 

0.14 

0.09 

2001  racial  distribution  of  students 

Percentage  of  black  students 

0.18 

0.03 

0.09 

0.59 

0.35 

Percentage  of  Hispanic  students 

0.00 

0.02 

0.04 

-7.31 

-16.083 

Percentage  of  American  Indian 
students 

0.00 

0.00 

0.00 

-0.11 

-0.16 

Percentage  of  Asian  students 

0.01 

0.01 

0.01 

-0.24 

-0.29 

Percentage  of  white  students 

0.81 

0.94 

0.86 

-0.53 

-0.21 

Transformed  percentage  of  white 
students 

0.34 

-0.09 

-0.22 

-0.37 

-0.18 

Percentage  low  income 

37.28 

25.65 

33.58 

0.53 

0.17 

Overall  attendance  rate 

93.52 

94.65 

94.04 

-0.73 

-0.34 

Overall  attendance  rate  information 
missing 

is  0.06 

0.07 

0.11 

-0.02 

-0.20 

Graduation  total  rate 

90.08 

90.92 

89.05 

-0.12 

0.15 
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Table  A. 39 — Continued 


Comparison  Group  Mean  Standardized  Bias 

MSP  Group - 

Covariate  Mean  Unweighted  Weighted  Unweighted  Weighted 


Median  absolute  standardized  bias — 2001  racial  distribution  of  students  0.45  0.21 

District  population,  2000  census 


Percentage  black  (population) 

0.09 

0.02 

0.05 

0.52 

0.30 

Percentage  white  (population) 

0.89 

0.95 

0.91 

-0.49 

-0.21 

Percentage  other  race  (population) 

0.02 

0.02 

0.03 

-0.14 

-0.97 

Percentage  urban 

0.78 

0.52 

0.72 

0.84 

0.18 

Percentage  unmarried  heads  of 
household  with  children 

0.29 

0.19 

0.26 

0.65 

0.22 

Percentage  of  female  heads  of 
household  with  children 

0.23 

0.14 

0.20 

0.69 

0.27 

Percentage  of  men  with  0-8  years  of 
education 

0.05 

0.06 

0.06 

-0.58 

-0.41 

Percentage  of  men  with  9-11  years  of 
education 

0.08 

0.10 

0.10 

-0.61 

-0.73 

Percentage  of  men  with  12  years  of 
education 

0.42 

0.45 

0.45 

-0.29 

-0.24 

Percentage  of  men  with  13-15  years  of 
education 

0.23 

0.20 

0.20 

0.82 

0.66 

Percentage  of  men  with  16+  years  of 
education 

0.22 

0.19 

0.19 

0.21 

0.22 

Percentage  of  women  with  0-8  years 
of  education 

0.06 

0.06 

0.06 

-0.18 

-0.36 

Percentage  of  women  with  9-11  years 
of  education 

0.09 

0.10 

0.10 

-0.31 

-0.63 

Percentage  of  women  with  12  years  of 
education 

0.45 

0.47 

0.47 

-0.32 

-0.27 

Percentage  of  women  with  13-15  years 
of  education 

0.23 

0.21 

0.20 

0.75 

0.77 

Percentage  of  women  with  16+  years 
of  education 

0.18 

0.17 

0.16 

0.08 

0.17 

Percentage  with  13+  years  of  education 

0.43 

0.38 

0.38 

0.38 

0.41 

Percentage  with  16+  years  of  education 

0.20 

0.18 

0.18 

0.14 

0.20 

Percentage  with  master's  degree 

0.05 

0.04 

0.04 

0.07 

0.14 

Percentage  of  men  not  in  the  labor 
force 

0.34 

0.30 

0.33 

0.54 

0.11 

Percentage  of  women  not  in  the  labor 
force 

0.48 

0.45 

0.47 

0.67 

0.23 

Percentage  of  population  age  16+  not 
in  the  labor  force 

0.41 

0.38 

0.40 

0.66 

0.18 
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Table  A. 39 — Continued 


MSP  Group 
Mean 

Comparison  Group  Mean 

Standardized  Bias 

Covariate 

Unweighted 

Weighted 

Unweighted 

Weighted 

Percentage  of  population  not  working 
in  Pennsylvania 

0.03 

0.05 

0.04 

-0.34 

-0.11 

Percentage  under  18 

0.23 

0.24 

0.23 

-0.54 

-0.22 

Percentage  over  65 

0.19 

0.16 

0.18 

0.74 

0.23 

Median  rent  ($) 

471 

500 

488 

-0.30 

-0.18 

Median  real  estate  tax  ($) 

1,516 

1,451 

1,425 

0.11 

0.16 

Median  home  value  ($) 

77,270 

97,010 

88,120 

-0.68 

-0.37 

Percentage  of  vacant  houses 

0.08 

0.11 

0.10 

-0.78 

-0.6 

Percentage  of  owner-occupied  houses 

0.71 

0.77 

0.72 

-0.57 

-0.09 

1999  median  household  income  ($) 

35,810 

41,120 

37,940 

-0.53 

-0.21 

1999  median  family  income  ($) 

44,680 

48,620 

46,010 

-0.34 

-0.12 

1999  per  capita  income  ($) 

19,410 

19,610 

19,130 

-0.04 

0.05 

Median  absolute  standardized  bias — district  population,  2000  census 

0.52 

0.22 

Population 

Population  per  square  mile 

1,844 

687 

1,402 

0.59 

0.23 

Transformed  population  per  square 
mile 

6.83 

5.43 

6.28 

1.06 

0.41 

Total  district  population 

18,280 

19,070 

23,700 

-0.07 

-0.48 

Median  absolute  standardized  bias — population 

0.59 

0.41 

2002  district  financial  data 

Average  daily  membership  (ADM) 

2,555 

2,899 

3,422 

-0.23 

-0.57 

Transformed  ADM 

2.04 

2.04 

2.06 

-0.02 

-0.32 

Percentage  of  students  in  families 
receiving  welfare 

0.06 

0.02 

0.04 

0.63 

0.26 

Pupil-teacher  ratio 

15.20 

15.70 

15.62 

-0.31 

-0.27 

Personal  income  tax  effort 

44.30 

43.65 

45.13 

0.06 

-0.08 

Transformed  personal  income  tax 
effort 

3.77 

3.74 

3.77 

0.10 

-0.03 

Average  teacher  salary  ($) 

49,570 

48,820 

50,130 

0.15 

-0.12 

Transformed  average  teacher  salary 

10.81 

10.79 

10.81 

0.19 

-0.09 

Ratio  of  district  market  value  to  total 
personal  income 

0.59 

0.55 

0.57 

0.20 

0.12 

Ratio  of  market  value  to  personal 
income  tax  effort 

24.18 

19.50 

21.40 

0.76 

0.45 

Transformed  ratio  of  market  value  to 

4.88 

4.39 

4.60 

0.81 

0.47 

income  tax  effort 
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Table  A. 39 — Continued 


MSP  Group 
Mean 

Comparison  Group  Mean 

Standardized  Bias 

Covariate 

Unweighted 

Weighted 

Unweighted 

Weighted 

Ratio  of  instructional  expense  to 

ADM  ($) 

5,888 

5,612 

5,738 

0.30 

0.17 

Transformed  ratio  of  instructional 
expense  to  ADM 

8.67 

8.62 

8.64 

0.00 

0.19 

Teacher  average  years  of  service 

15.97 

16.66 

16.91 

-0.31 

-0.43 

Median  absolute  standardized  bias — 2002  district  financial  data 

0.22 

0.23 

Interactions 

Interaction:  ADM,  ratio  of  market  value 
to  income 

1421 

1443 

1882 

-0.03 

-0.57 

Interaction:  ADM,  tax  effort 

60,550 

59,010 

77,390 

0.04 

-0.46 

Interaction:  ADM,  pupil-teacher  ratio 

39,520 

46,610 

54,600 

-0.29 

-0.62 

Interaction:  ADM,  population  per 
square  mile 

4,387,000 

3,266,000 

7,082,000 

0.21 

-0.51 

Median  absolute  standardized  bias — interactions 

0.13 

0.54 

Median  absolute  standardized  bias — all  covariates 

0.24 

0.16 

NOTE:  Comparison  group  3  (n  =  338  school  districts)  consists  of  non-MSP  Pennsylvania  school  districts,  excluding 
Pittsburgh  and  Philadelphia  city  school  districts,  a  district  with  no  enrolled  students,  districts  that  participated 
in  the  Philadelphia  MSP,  and  districts  that,  though  not  members  of  the  MSP,  participated  in  some  MSP  activities. 
The  MSP  group  (n  =  48  school  districts)  excludes  six  MSP  districts  that  did  not  join  the  project  until  2006. 

a  The  standardized  bias  of  this  covariate  is  inflated  because  there  is  little  variation  within  the  MSP  group;  using 
the  standard  deviation  across  all  districts  produces  a  weighted  standardized  bias  of  -0.64. 
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Table  A.40 

State-Level  Regression  Results  for  Mathematics,  Cohort  G  (Grade  11  in  2006-07) 


Comparison  Group  1  Comparison  Group  2  Comparison  Group  3 


Variable 

Coef. 

Std. 

Err. 

t 

p-value 

Coef. 

Std. 

Err. 

t 

p-value 

Coef. 

Std. 

Err. 

t 

p-value 

Math  proficiency 
levels  in  2003-04 
(grade  8) 

0.63 

0.08 

8.03 

0.00 

0.63 

0.08 

7.64 

0.00 

0.65 

0.08 

7.66 

0.00 

Hispanic  percentage 
of  enrollment 

-33.6 

27.3 

-1.23 

0.22 

-37.7 

30.6 

-1.23 

0.22 

-27.6 

30.3 

-0.91 

0.36 

Black  percentage  of 
enrollment 

-8.89 

7.52 

-1.18 

0.24 

-8.87 

7.98 

-1.11 

0.27 

-7.90 

7.96 

-0.99 

0.32 

Other  race/ethnicity 
percentage  of 
enrollment 

12.2 

53.9 

0.23 

0.82 

6.64 

59.0 

0.11 

0.91 

0.14 

55.1 

0.00 

1.00 

Percentage  of 
males  in  community 
educated  9-11  years 

-119.0 

37.2 

-3.19 

0.00 

-130.0 

36.1 

-3.60 

0.00 

-133. 

39.5 

-3.37 

0.00 

Percentage  of 
males  in  community 
educated  13-15  years 

-12.9 

27.3 

-0.47 

0.64 

-15.9 

28.0 

-0.57 

0.57 

5.36 

32.5 

0.16 

0.87 

Percentage  of 
females  in  community 
educated  9-11  years 

41.0 

40.6 

1.01 

0.32 

47.0 

40.4 

1.17 

0.24 

45.5 

45.5 

1.00 

0.32 

Percentage  of 
females  in  community 
educated  13-15  years 

-32.2 

39.9 

-0.81 

0.42 

-33.1 

41.6 

-0.80 

0.43 

-52.6 

43.4 

-1.21 

0.23 

Teacher  years  of 
service 

0.13 

0.33 

0.40 

0.69 

0.12 

0.34 

0.35 

0.73 

0.29 

0.35 

0.83 

0.41 

Interaction  of  average 
daily  enrollment  and 
pupil-teacher  ratio 

0.00 

0.00 

1.16 

0.25 

0.00 

0.00 

1.34 

0.18 

0.00 

0.00 

0.71 

0.48 

Leader  participation 

0.68 

0.76 

0.89 

0.37 

0.51 

0.79 

0.65 

0.52 

0.75 

0.86 

0.88 

0.38 

Math  teacher 
participation 

2.63 

2.18 

1.21 

0.23 

2.74 

2.24 

1.22 

0.22 

2.76 

2.45 

1.13 

0.26 

Science  teacher 
participation 

-3.18 

2.34 

-1.36 

0.18 

-3.19 

2.42 

-1.32 

0.19 

-3.39 

2.64 

-1.28 

0.20 

NOTE:  Comparison  group  1  (n  =  442  school  districts)  consists  of  non-MSP  Pennsylvania  school  districts,  excluding 
Pittsburgh  and  Philadelphia  city  school  districts  and  a  district  with  no  enrolled  students.  Comparison  group  2 
(n  =  412  school  districts)  excludes  districts  from  group  1  that  participated  in  the  Philadelphia  math  and  science 
partnership.  Comparison  group  3  (n  =  338  school  districts)  excludes  districts  from  group  2  that,  though  not 
members  of  the  MSP,  participated  in  some  MSP  activities.  The  MSP  group  (n  =  48  school  districts)  excludes  six  MSP 
districts  that  did  not  join  the  project  until  2006.  District  covariates  included  in  models  are  those  among  the  106 
pre-MSP  covariates  used  in  the  propensity-weight  process  that  did  not  balance  well  (p  <  0.05,  that  differences 
between  the  MSP  group  and  weighted  comparison  group  were  due  to  chance). 
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Table  A.41 

State-Level  Regression  Results  for  Mathematics,  Cohort  J  (Grade  8  in  2006-07) 


Comparison  Group  1  Comparison  Group  2  Comparison  Group  3 


Variable 

Coef. 

Std. 

Err. 

t 

p-value 

Coef. 

Std. 

Err. 

t 

p-value 

Coef. 

Std. 

Err. 

t 

p-value 

Math  proficiency 
levels  in  2003-04 
(grade  5) 

0.32 

0.08 

4.14 

0.00 

0.31 

0.08 

3.95 

0.00 

0.33 

0.07 

4.55 

0.00 

Hispanic  percentage 
of  enrollment 

-18.8 

24.2 

-0.77 

0.44 

-14.0 

30.2 

-0.46 

0.64 

-37.5 

29.2 

-1.28 

0.20 

Black  percentage  of 
enrollment 

-45.1 

7.1 

-6.35 

0.00 

-44.8 

7.22 

-6.21 

0.00 

-46.0 

7.67 

-6.00 

0.00 

Other  race/ethnicity 
percentage  of 
enrollment 

77.4 

57.8 

1.34 

0.18 

69.2 

61.9 

1.12 

0.26 

99.5 

65.7 

1.52 

0.13 

Percentage  of 
males  in  community 
educated  9-11  years 

-54.4 

36.5 

-1.49 

0.14 

-60.9 

35.8 

-1.70 

0.09 

-52.9 

35.2 

-1.50 

0.13 

Percentage  of 
males  in  community 
educated  13-15  years 

9.18 

32.8 

0.28 

0.78 

7.32 

34.3 

0.21 

0.83 

8.82 

32.6 

0.27 

0.79 

Percentage  of 
females  in  community 
educated  9-11  years 

-54.4 

47.5 

-1.14 

0.25 

-54.9 

48.7 

-1.13 

0.26 

-36.8 

45.4 

-0.81 

0.42 

Percentage  of 
females  in  community 
educated  13-15  years 

7.33 

41.6 

0.18 

0.86 

2.89 

44.4 

0.07 

0.95 

25.3 

39.8 

0.64 

0.52 

Teacher  years  of 
service 

-0.54 

0.39 

-1.38 

0.17 

-0.58 

0.40 

-1.44 

0.15 

-0.43 

0.41 

-1.04 

0.30 

Interaction  of  average 
daily  enrollment  and 
pupil-teacher  ratio 

0.00 

0.00 

-2.36 

0.02 

0.00 

0.00 

-2.00 

0.05 

0.00 

0.00 

-2.10 

0.04 

Leader  participation 

2.79 

0.97 

2.87 

0.00 

2.83 

1.01 

2.79 

0.01 

2.79 

1.07 

2.62 

0.01 

Math  teacher 
participation 

0.76 

1.96 

0.39 

0.70 

0.77 

2.03 

0.38 

0.70 

0.89 

2.21 

0.40 

0.69 

Science  teacher 
participation 

-4.59 

2.34 

-1.96 

0.05 

-4.65 

2.42 

-1.92 

0.06 

-5.18 

2.63 

-1.97 

0.05 

NOTE:  Comparison  group  1  (n  =  442  school  districts)  consists  of  non-MSP  Pennsylvania  school  districts,  excluding 
Pittsburgh  and  Philadelphia  city  school  districts  and  a  district  with  no  enrolled  students.  Comparison  group  2 
(n  =  412  school  districts)  excludes  districts  from  group  1  that  participated  in  the  Philadelphia  math  and  science 
partnership.  Comparison  group  3  (n  =  338  school  districts)  excludes  districts  from  group  2  that,  though  not 
members  of  the  MSP,  participated  in  some  MSP  activities.  The  MSP  group  (n  =  48  school  districts)  excludes  six  MSP 
districts  that  did  not  join  the  project  until  2006.  District  covariates  included  in  models  are  those  among  the  106 
pre-MSP  covariates  used  in  the  propensity-weight  process  that  did  not  balance  well  (p  <  0.05,  that  differences 
between  the  MSP  group  and  weighted  comparison  group  were  due  to  chance). 
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This  appendix  contains  information  about  the  survey  development  and  analysis  related  to  the 
findings  reported  in  Chapter  Five. 


Teacher  Survey 

To  develop  the  teacher  survey,  we  reviewed  existing  survey  instruments  from  several  related 
studies  of  instructional  practice,  such  as  Horizon  Research’s  Looking  Inside  the  Classroom 
(I.  Weiss,  Pasley,  et  ah,  2003),  RAND’s  Mosaic  II  project,  instruments  from  Michigan  State 
University’s  TIMSS  Study  Center,  and  the  Council  of  Chief  State  School  Officers’  Surveys  of 
Enacted  Curriculum.  We  selected  the  Surveys  of  Enacted  Curriculum  based  on  their  ability  to 
measure  the  types  of  teacher  practice  and  course  content  reforms  targeted  by  the  MSP  inter¬ 
vention  (Porter,  2002).  However,  because  these  surveys  must  be  administered  near  the  end  of 
the  academic  year  and  require  60-90  minutes  for  completion,  we  were  concerned  about  our 
ability  to  achieve  acceptable  response  rates  with  the  instruments  in  their  current  form. 

In  consultation  with  colleagues,  we  determined  that  30-45  minutes  was  a  more  reason¬ 
able  length  for  this  survey.  Therefore,  we  modified  the  survey  by  deleting  items  so  that  the 
expected  completion  time  would  fall  into  this  range.  The  criteria  we  used  to  determine  which 
items  to  delete  were  based  on  the  relevance  of  each  survey  item  to  the  MSP  intervention.  We 
preserved  those  items  that  we  expected  to  be  affected  most  strongly  by  the  project.  A  pilot  test 
was  conducted  to  ensure  that  the  modified  instrument  maintained  coherence  and  could  be 
completed  in  the  targeted  amount  of  time.  Our  adaptation  of  the  survey  instrument  was  cre¬ 
ated  with  permission  of  the  survey  developers. 

The  teacher  population  of  interest  for  our  survey  was  all  teachers  in  the  original  40  MSP 
districts  who  specialize  in  math  or  science  or  who  teach  at  least  one  math  or  science  class. 
At  the  elementary-school  level,  this  meant  that  the  population  often  included  most  teach¬ 
ers  in  each  school.  We  drew  a  stratified  random  sample  of  the  population  for  inclusion  in  the 
survey,  oversampling  teachers  involved  in  the  MSP  LATs.  Elementary-school  teachers  sampled 
to  answer  the  survey  were  randomly  assigned  either  the  math  or  science  survey  if  they  taught 
both  subjects. 

The  survey  was  conducted  online,  with  a  paper  survey  available  to  teachers  who  were 
unable  or  unwilling  to  complete  the  survey  online.  Sample  and  response  rates  are  presented  in 
Table  B.l. 


ill 
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Table  B.1 

Teacher  Survey  Sample  and  Response  Rates 


Survey 

Responses 

Sample 

Total  Population 

Response  Rate  (%) 

Baseline 

1,241 

1,881 

3,200 

66 

Follow-up 

1,574 

1,988 

3,440 

79 

Principal  Survey 

The  principal  survey  was  designed  to  capture  changes  in  principals’  views  and  attitudes  toward 
science  and  mathematics  instruction;  current  practices  and  policies  regarding  curriculum, 
instruction,  assessment,  and  professional  development;  district  and  IU  support  for  improving 
schools;  and  MSP  project  impact.  Survey  items  were  adapted  from  survey  instruments  devel¬ 
oped  by  Horizon  Research,  Inc.,  the  Center  for  the  Study  of  Teaching  and  Policy,  and  the 
Center  for  Research  on  the  Context  of  Teaching  and  from  principal  rubrics  developed  by  Rich¬ 
ard  Halverson  at  the  Wisconsin  Center  for  Education  Research.  We  also  shared  early  drafts  of 
our  principal  survey  with  the  MSP  PI  and  co-PI  and  received  valuable  input  on  refining  the 
survey  items  to  reflect  areas  of  MSP  impact. 

The  survey  consisted  of  six  sections.  The  first  two  sections  focused  on  views  and  influences 
on  mathematics/science  instructional  practices  and  asked  principals  to  provide  their  opinions 
about  lesson  design  and  implementation;  instructional  practices;  classroom  resources;  assess¬ 
ment  of  student  learning;  government  (federal  and  state),  regional/district,  and  school  policies; 
instructional  materials  and  resources;  and  teacher  support.  The  third  section  included  the  most 
diverse  set  of  survey  items,  with  questions  focusing  on  principal  practices  related  to  curricu¬ 
lum,  instruction,  and  assessment,  with  a  number  of  questions  about  the  types  of  professional 
development  activities  in  which  the  principals  participate,  including  those  offered  by  the  MSP. 
The  fourth  section,  on  district  and  IU  support  for  improving  schools,  solicited  principals’  views 
on  the  extent  to  which  districts  and  IUs  facilitate  supportive  environments  for  many  of  the 
MSP-promoted  practices.  Principals  were  also  asked  to  rate  the  MSP  staff’s  role  in  supporting 
school  efforts  to  implement  MSP  practices,  such  as  data  collection  and  analysis,  development 
of  standards,  and  staff  development.  The  fifth  section  focused  solely  on  MSP  project  impact, 
asking  principals  to  judge  the  impact  of  activities  included  in  the  MSP  action  plan.  Finally,  the 
sixth  section  collected  information  about  the  principals’  background  and  included  items  on 
content  expertise,  years  as  principal/assistant  principal,  ethnicity,  and  academic  degree. 

The  baseline  principal  survey  was  fielded  online  in  late  2004,  with  paper  surveys  avail¬ 
able  to  principals  who  requested  them.  The  survey  was  distributed  to  201  principals  (one  from 
each  school  in  the  48  MSP  districts — NSF-  and  PDE-funded).  The  overall  response  rate  was 
71  percent,  and  11  percent  of  these  responses  were  on  paper.  The  response  rate  for  the  follow-up 
survey,  fielded  in  late  2006,  was  92  percent  (181  of  197).  Surveys  were  received  in  both  years 
from  129  schools.  From  2004  to  2006  there  was  considerable  turnover  among  principals  in  the 
MSP  schools.  As  a  result,  the  follow-up  sample  included  only  118  of  the  201  principals  who 
had  been  sampled  in  2004,  and  84  of  these  responded  to  both  surveys. 

Notably,  the  turnover  among  principals  resulted  in  a  shift  in  some  of  the  demographic 
variables  summarized  in  Table  B.2.  In  general,  principals  responding  in  2006  reported 
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Table  B.2 

Principal  Survey  Respondent  Demographics 


Characteristic 

2004  Survey  (%) 

2006  Survey  (%) 

Gender 

Female 

46 

47 

Race 

White 

90 

91 

Black 

6 

7 

Years  as  principal  in  the  district 

>  10 

19 

9 

7-10 

24 

20 

4-6 

26 

29 

2-3 

19 

24 

0-1 

11 

19 

Highest  degree  earned 

Doctorate 

28 

21 

Master's  + 

29 

35 

Master's 

42 

43 

fewer  years  of  experience  in  their  district  than  did  those  who  responded  in  2004  (Cochran- 
Armitage  Trend  Test,  p  <  0.01).  In  2006,  9  percent  of  principals  reported  having  more  than 
10  years  of  experience  as  a  principal  in  the  district,  while  19  percent  reported  having  one  year 
or  less.  In  2004,  these  values  were  approximately  reversed:  Nineteen  percent  reported  having 
more  than  10  years  of  experience,  while  11  percent  reported  having  one  year  or  less.  A  similar 
trend,  though  nonsignificant,  appears  in  the  educational  degrees  reported,  with  fewer  prin¬ 
cipals  in  2006  holding  a  doctorate  degree  and  more  with  a  master’s  degree  plus  additional 
coursework. 


Factor  Analysis  of  Principal  and  Teacher  Survey  Data 

The  principal  surveys  consisted  of  sections  pertaining  to  views  and  influences  on  mathemat¬ 
ics  and  science  instruction;  current  practices  and  policies  regarding  curriculum,  instruction, 
assessment,  and  professional  development;  and  district  and  IU  support  for  improving  schools. 
Each  section  included  questions  that  had  between  four  and  25  subquestions  using  Likert-type 
scales  (e.g.,  1  =  strongly  disagree,  5  =  strongly  agree).  The  teacher  surveys  consisted  of  sections 
pertaining  to  instructional  activities,  assessments,  instructional  influences,  classroom  prepara¬ 
tion,  and  professional  development  for  math  and  science  in  high  school  or  K-8.  Each  section 
included  questions  that  had  between  five  and  12  subquestions.  The  goal  was  to  develop  scales 
that  would  summarize  these  sections. 
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Factor  analysis  was  used  to  determine  the  grouping  of  variables  in  each  section.  To  be 
more  specific,  factor  analysis  was  used  to  uncover  the  latent  structure  (dimensions)  of  a  set  of 
variables.  In  essence,  we  used  the  factor  analysis  for  the  following  purposes: 

•  to  select  a  subset  of  variables  from  a  larger  set,  based  on  which  original  variables  have  the 
highest  correlations  with  the  principal  component  factors 

•  to  validate  a  scale  or  index  by  demonstrating  that  its  constituent  items  load  on  the  same 
factor  and  to  drop  proposed  scale  items  that  cross-load  on  more  than  one  factor. 

After  factor  analysis,  a  descriptive  name  was  attached  to  each  common  factor  once  it  was 
extracted  and  identified.  The  assigned  name  is  indicative  of  the  predominant  concern  that  each 
factor  addresses.  At  this  point,  the  named  common  factors  can  now  be  used  as  independent  or 
predictor  variables.  However,  a  reliability  test  is  meaningful  for  all  the  factors  before  they  are 
used  in  subsequent  analyses. 

Reliability  comes  to  the  forefront  when  variables  developed  from  summated  scales  are 
used  as  predictor  components  in  objective  models.  Because  summated  scales  are  an  assem¬ 
bly  of  interrelated  items  designed  to  measure  underlying  constructs,  it  is  very  important  to 
know  whether  the  same  set  of  items  would  elicit  the  same  responses  if  the  same  questions  were 
recast  and  readministered  to  the  same  respondents.  Variables  derived  from  test  instruments 
are  declared  reliable  only  when  they  provide  stable  responses  over  repeated  administration  of 
the  test. 

To  test  the  reliability  of  the  factors  (or  scales),  Cronbach’s  alpha  was  calculated.  Computa¬ 
tion  of  alpha  is  based  on  the  reliability  of  a  test  relative  to  other  tests  with  same  number  of  items 
and  measuring  the  same  construct  of  interest  (Hatcher,  1994).  The  alpha  coefficient  ranges  in 
value  from  0  to  1.  The  higher  the  score,  the  more  reliable  the  generated  scale.  Nunnally  (1978) 
has  found  0.7  to  be  an  acceptable  reliability  coefficient,  but  lower  thresholds  are  sometimes 
used  in  the  literature.  For  nearly  all  of  our  scales,  alpha  is  >  0.60  (as  reported  in  Tables  5.1  and 
5.2  in  Chapter  Five).  For  each  scale,  the  dispensable  variable  or  variables  were  identified  by 
listing  the  deleted  variable  with  the  expected  resultant  alpha.  If  alpha  was  improved,  then  the 
dispensable  variables  were  removed  from  the  scale.  In  other  words,  the  removing  of  a  variable 
from  the  scale  will  make  the  construct  more  reliable  for  use  as  a  predictor  variable. 

Once  a  scale  was  established,  tests  for  normality  (including  the  Shapiro-Wilk  test)  were 
performed.  When  scales  were  normally  distributed,  comparisons  between  groups  were  made 
using  a  t-test  or  ANOVA  (analysis  of  variance),  where  applicable.  Where  the  distributions  were 
not  normal,  comparisons  between  groups  were  made  using  a  Kruskal-Wallis  test  or  signed- 
rank  test,  where  applicable.  In  all  cases,  p-values  less  than  0.05  were  considered  evidence  of 
differences  not  attributable  to  chance.  All  analyses  were  performed  with  use  of  SAS®  9.1  (SAS 
Institute,  2004). 


Imputation  of  Missing  Values 

For  each  of  the  four  surveys  (baseline  and  follow-up,  teacher  and  principal),  we  imputed 
responses  that  were  missing  on  surveys  that  we  received.  Generally,  there  were  few  missing 
items.  Additionally,  we  imputed  response  values  for  the  entire  survey  for  nonrespondents  on 
the  baseline  and  follow-up  principal  surveys  and  on  the  follow-up  teacher  surveys.  The  response 
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rates  for  those  surveys  were  discussed  earlier  in  this  appendix;  each  was  71  percent  or  greater. 
The  imputations  enabled  us  to  make  inferences  about  the  full  population  of  principals,  and,  in 
combination  with  sample  weighting,  enabled  us  to  make  inferences  about  the  full  population 
of  teachers.  Multiple  (five)  imputations  of  missing  values  were  performed  using  a  multivariate 
technique  consisting  of  a  sequence  of  stepwise  regression  models.  Imputations  were  performed 
separately  for  each  year  and  each  subject  (math  and  science).  Possible  predictor  variables  in  each 
model  consisted  of  all  item-level  responses  for  a  given  year,  as  well  as  teacher,  school,  and  dis¬ 
trict  demographics.  To  check  that  the  imputation  models  performed  reasonably  in  recovering 
the  missing  data,  we  examined  the  pre-  and  post-imputation  data  to  confirm  that  the  imputa¬ 
tion  did  not  produce  unexpectedly  large  changes  to  item-response  patterns.  For  example,  on 
the  follow-up  teacher  survey  for  mathematics  teachers,  out  of  148  survey  items,  the  absolute 
difference  in  item  means  from  pre-  to  post-imputation  averaged  0.08  points  on  the  survey 
scale,  and  the  greatest  change  in  mean  for  any  single  item  was  0.26  points.  All  subsequent 
analyses  of  the  imputed  data  adjust  for  the  between-  and  within-imputation  variability — that 
is,  all  subsequent  statistical  tests  account  for  the  uncertainty  introduced  by  the  imputation. 


Statistical  Models  for  Survey  Analysis 

We  ran  hierarchical  linear  models  that  account  for  the  clustering  of  teachers  in  schools  and 
districts  to  assess  how  responses  changed.  In  all  cases,  two-sided  tests  were  used  with  p-values 
for  rejecting  the  null  hypothesis  set  lower  than  0.05  to  correct  for  multiple-hypothesis  testing. 
Here,  we  describe  the  statistical  models  used  in  the  analysis  of  the  teacher  and  principal  survey 
data. 

Teacher  Survey 

Teacher  survey  data  were  analyzed  separately  for  each  subject  (mathematics  and  science)  and 
grade  level  (high-school  survey  or  K-8  survey).  We  stratified  on  these  variables  to  avoid  bias 
that  could  be  introduced  from  the  sampling  mechanism.  The  mixed  model  used  to  analyze  the 
data  can  be  broken  down  into  two  levels: 


Y. 


B  +  r... 

0 1  v 


This  equation  models  teacher-level  effects,  where  BQ.  is  the  intercept  for  the  teacher’s 
school,  and  r  is  the  random  effect  for  zth  teacher  at  the 7th  school.  The  within-school  random 
error,  r  ,  is  considered  to  be  normally  distributed  with  mean  zero  and  variance  0~ .  The 
outcome,  Y  ,  is  the  difference  in  the  zth  standardized  teacher  survey  scale  between  2007  and 
2004  in  the  yth  school.  Both  the  2004  and  2007  survey  scales  were  standardized  using  the 
standard  deviation  for  2004.  Twelve  survey  scales  and  all  individual  items  in  the  assessment 
scale  were  analyzed  in  separate  models.  The  second  level  of  the  model  is  as  follows: 

B0j  =  7oo  +  Y01  [pn )  +  Y02  {p,2  )  +  7o3  (p7-3  )  +  p0j  • 


This  breaks  down  the  intercept  term  for  each  school,  BQ  .  It  includes  a  term  for  the  over¬ 
all  mean,  yoo,  and  three  school-level  fixed  effects:  yoi,  yo2,  and  yo3,  representing  the  effects  of 
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MSP  participation  of  district-level  leaders,  school-level  leaders,  and  teachers.  Participation  by 
these  educators  at  school  j  is  represented  by  P  t ,  P.  , ,  and  P  ,  respectively.  Teacher  participa¬ 
tion  was  subject-specific  for  the  analyses  performed  on  higA-school  teachers,  but  for  the  K-8 
teacher  analysis,  the  participation  of  teachers  in  both  subjects  was  combined  because  it  is 
common  for  teachers  of  lower  grade  levels  to  teach  both  subjects.  The  participation  variables 
were  a  cumulative  sum  of  participation  in  MSP  through  the  2006-07  school  year.  The  vari¬ 
ance  of  the  among-school  random  effect,  /J-Q. ,  is  modeled  as  having  a  normal  distribution  with 
mean  of  zero  and  a  variance  of  T2. 

Each  imputation  dataset  was  tested  separately  and  combined  using  PROC  MIANALYZE 
in  SAS.  In  considering  the  multiple  tests  performed  for  each  scale  of  the  teacher  survey,  we 
applied  a  Bonferroni  least  significant  difference  adjustment  to  the  overall  significance  level  of 
the  F  tests,  dividing  the  significance  level  of  0.05  by  the  number  of  tests,  /. 

Principal  Survey 

The  modeling  approach  used  in  the  principal  survey  was  again  a  mixed  model,  this  time 
including  only  participation  data  for  school  leaders  at  the  district  and  school  levels  as  well  as  a 
variable  indicating  the  number  of  school  leaders  participating  in  principal  seminars.  The  par¬ 
ticipation  variables  are  cumulative  sums  of  participation  hours  through  the  2005-06  school 
year.  The  mixed  model  used  to  analyze  the  data  is  as  follows: 


Level  1 :  Y.  =  B  1  +  r.. 

9  0  9 

Level  2:  By  =  y10  +  7„  [PjX )  +  712  (P2 )  +  7I3  [PjP]  +  Ptj  > 

where  changes  in  standardized  survey  scales  from  2004  to  2006,  Y  ,  were  modeled  as  a  linear 
combination  of  Bx.,  the  intercept  for  the  principal’s  school,  and  r  ,  the  random  effect  for  zth 
principal  at  the^'th  school,  r  is  normally  distributed  with  mean  zero  and  variance  0~ .  Seven¬ 
teen  survey  scales  were  analyzed  in  separate  models.  The  intercept  term  for  each  school,  B{., 
is  modeled  as  a  linear  combination  of  the  overall  mean,  y  ;  two  school-level  participation 
fixed  effects,  yn  and  yi3,  represent  MSP  participation  of  school  leaders  at  the  district  t  j  and 
school  [Pj2 )  levels;  and  a  third  school-level  fixed  effect,  yi3,  is  a  count  of  educators  in  the  school 
who  attended  principal  seminars  The  variance  of  the  school  random  effect,  pxj,  is  mod¬ 

eled  as  having  a  normal  distribution  with  mean  of  zero  and  a  variance  of  T2. 

Each  imputation  dataset  was  tested  separately  and  combined  using  PROC  MIANALYZE 
in  SAS.  For  each  family  of  tests  of  size  n,  a  least  significant  difference  adjustment  of  0.05 In  was 
performed  to  the  overall  level  of  significance. 


APPENDIX  C 


Qualitative  Analysis  of  IHE  Data 


This  appendix  contains  additional  information  about  the  methods  used  for  the  qualitative 
analyses  of  IHE  data  reported  in  Chapters  Five  and  Six. 


Data  Sources 

The  primary  data  source  for  the  IHE  qualitative  analyses  was  semistructured  individual  inter¬ 
views  with  IHE  STEM  and  education  faculty  members;  administrators,  including  deans, 
department  administrators,  and  coordinators  of  student-teacher  placements;  student  teachers; 
and  K-12  teachers  participating  in  the  TF  program.  From  2004  through  2008,  we  conducted 
interviews  of  56  IHE-affiliated  faculty  members  and  administrators,  including  deans,  depart¬ 
ment  administrators,  and  student-teacher  placement  coordinators.  We  used  purposive  sampling 
to  gather  information  from  STEM  faculty  members  in  math  and  science  and  from  science  and 
math  education  faculty  members  as  well  as  to  select  the  faculty  members  who  were  interviewed 
more  than  once.  In  addition,  we  interviewed  10  TFs  (at  least  two  from  each  IHE),  four  stu¬ 
dent  teachers,  and  four  K-12  teachers  who  supervised  student  teachers.  In  sum,  we  conducted 
a  total  of  118  interviews.  In  addition  to  the  individual  interviews,  we  conducted  three  focus 
groups  with  TFs  who  had  completed  the  TF  program.  Supporting  data  sources  consisted  of 
four  IHE  classroom  observations,  documentation  review  of  IHE  course  syllabi  and  curricula, 
questionnaires  regarding  MSP  course  revision,  MSP  and  K-20  partnership  literature,  and  the 
NSF’s  Management  Information  System,  a  database  of  information  from  annual  surveys  of  all 
Math  and  Science  Partnership  projects. 


Analysis 

Interviewer  notes  from  the  interviews  and  focus  groups  were  analyzed  using  the  qualitative 
data  management  software,  ATLAS.ti  (Muhr,  2004).  For  the  focus  group  data,  recordings  of 
the  discussions  were  archived  as  a  reference  from  which  to  draw  direct  quotations  for  the  analy¬ 
sis.  Questions  from  the  interview  protocol  provided  the  basis  for  the  preliminary  list  of  codes, 
which  was  refined  based  on  input  from  interviewers  and  new  themes  found  in  the  interviews. 
Two  researchers  worked  together  to  develop  the  final  code  list,  and  one  researcher  was  respon¬ 
sible  for  the  coding  to  ensure  the  consistent  application  of  codes  to  the  data. 

For  analyses  related  to  partnership  conceptualization  and  development,  three  sources  of 
data  were  used  to  build  construct  validity  and  the  reliability  of  findings  and  conclusions  (Yin, 
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2003):  (1)  MSP  and  K-20  partnership  literature,  (2)  semistructured  interviews,  and  (3)  exist¬ 
ing  MSP  Management  Information  System  online  survey  reports  and  annual  project  reports. 
MSP  and  K-20  partnership  literature  was  used  to  build  the  theoretical  framework  necessary 
to  interpret  the  empirical  data,  to  inform  code  development,  and  to  support  or  challenge  our 
findings.  The  first  mode  of  inquiry  that  we  employed  in  this  study  was  a  review  of  the  MSP  and 
K-20  literature  to  establish  a  theoretical  framework  that  allowed  us  to  operationalize  partner¬ 
ship  and  partnership  development.  However,  the  main  methodology  employed  in  this  study  was 
qualitative  data  analysis.  Using  existing  MSP  and  K-20  partnership  literature  and  our  research 
questions,  we  developed  a  list  of  descriptive  and  pattern  codes.  This  a  priori  list  of  “parent” 
codes  included  such  codes  as  institutional  factors,  MSP  impact,  partnership,  and  sustainabil¬ 
ity.  The  parent  codes  provided  an  organizational  structure  that  enabled  us  to  expand  these 
codes  into  subcodes  as  needed.  Descriptive  accounts  were  then  coded  to  create  a  systematic 
way  to  group  like  descriptions  of  partnership  development  and  to  compare  different  descrip¬ 
tions  of  partnership  development. 

The  first  iteration  of  coding  was  descriptive:  The  entire  interview  was  coded  with  parent 
and  subcodes,  allowing  us  to  classify  and  count  various  responses  related  to  our  research  ques¬ 
tions.  The  second  iteration  of  coding  was  more  interpretive:  We  attempted  to  identify  patterns 
and  explanations.  For  this  level  of  analysis,  we  focused  primarily  on  the  interview  segments 
that  were  coded  with  the  institutional  factor  and  partnership  parent  codes.  Focusing  on  just 
these  data,  we  coded  for  partnership  conceptualizations  and  factors  that  indicated  partnership 
development  based  on  our  review  of  the  literature.  Throughout  the  data-analysis  process,  we 
looked  for  disconfirming  and  confirming  evidence  to  create  an  accurate  and  nuanced  account 
of  partnership  development  (Erikson,  1986).  Interviews  were  coded  until  codes  were  saturated 
and  sufficient  numbers  of  regularities  emerged  (Miles  and  Huberman,  1994). 

We  analyzed  both  IHE  faculty  and  K-12  educator  interviews.  Data  were  collected  on  42 
IHE  faculty  (STEM  and  education)  and  10  K-12  educators  from  all  four  IHEs  between  2004 
and  2007.  Although  many  of  the  42  IHE  faculty  members  were  interviewed  repeatedly  over 
the  three  years  of  data  collection,  we  included  only  the  most  recent  interview  for  each  IHE 
faculty  member  in  our  sample.  The  most  recent  interviews  were  analyzed  because  conceptual¬ 
izations  of  partnership  and  beliefs  about  partnership  development  from  these  interviews  had 
the  most  relevance  with  regard  to  sustaining  the  project.  In  sum,  eight  IHE  faculty  interviews 
were  analyzed  from  year  2  (2004-05),  seven  were  analyzed  from  year  3  (2005-06),  and  27 
were  analyzed  from  year  4  (2006-07). 

The  10  interviews  with  K-12  educators  who  had  participated  in  the  TF  program  were 
also  analyzed.  The  TF  program  is  a  unique  program  that  provides  K-12  educators  an  opportu¬ 
nity  to  spend  one  or  two  semesters  at  a  partner  IHE,  working  alongside  faculty  to  revise  under¬ 
graduate  courses,  take  college  courses,  and  assist  in  other  MSP  activities.  Four  K-12  educator 
interviews  were  analyzed  from  year  2  (2004-05),  four  were  analyzed  from  year  3  (2005-06), 
and  two  were  analyzed  from  year  4  (2006-07). 

Finally,  we  examined  the  Management  Information  System  and  previous  annual  project 
reports  to  gather  participation  and  demographic  information  about  our  sample.  Our  review 
of  the  MSP  and  K-20  partnership  literature  suggested  there  were  at  least  five  characteristics 
that  might  influence  faculty  participation  and  attitudes  about  partnership  development:  IHE 
faculty  department,  IHE  faculty  tenure  status,  MSP  activity  participation,  IHE  faculty  par¬ 
ticipation  level  throughout  the  MSP,  and  gender.  We  found  our  sample  to  be  fairly  evenly  dis¬ 
tributed  across  these  five  characteristics  (see  Table  C.l). 
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Table  C.1 

IHE  Faculty  Member  Characteristics 


Characteristics 

Number  of 
interviewees 

Department 

Mathematics 

10 

Science 

16 

Education 

10 

Education  and  mathematics 

4 

Education  and  science 

2 

Tenure  status 

Tenured 

16 

Not  tenured 

17 

Not  applicable 

9 

Participation  level 

>160  hours 

8 

81-160  hours 

10 

20-80  hours 

13 

<  20  hours 

9 

Not  applicable 

2 

Gender 

Male 

23 

Female 

19 

Classroom  Observation  Methodology 

Between  spring  2007  and  spring  2008,  a  total  of  four  IHE  classroom  observations  were 
conducted — one  at  each  of  the  four  IHEs.  The  primary  goal  of  the  classroom  observations 
was  to  provide  a  more  in-depth  description  of  the  changes  in  instructional  practices  reported 
by  IHE  faculty.  However,  a  secondary  goal  of  the  classroom  observation  was  to  gain  a  better 
understanding  of  the  nature  of  the  course  revisions.  Thus,  we  specifically  asked  to  observe 
courses  that  had  been  revised  by  TFs.  For  all  except  one  of  the  observations,  we  were  able 
to  observe  a  lesson  from  a  course  that  had  been  revised  by  a  TF.  The  observational  protocol 
that  we  developed  consisted  of  a  set  of  items  that  asked  the  observers  to  describe  the  types  of 
instruction  implemented  by  the  IHE  faculty,  the  level  of  student  involvement,  and  student  cog¬ 
nitive  activity.  These  items  were  drawn  from  the  classroom  observation  protocol  developed  for 
the  NSF’s  Collaboratives  for  Excellence  in  Teacher  Preparation  program  (Lawrenz,  Huffman, 
and  Appledoorn,  2002). 
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We  observed  a  single  class  session  at  each  of  the  IHEs;  overall,  these  sessions  included 
one  introductory  science  class,  one  advanced-level  science  class,  one  introductory  mathemat¬ 
ics  class,  and  one  advanced-level  mathematics  class.  We  interviewed  IHE  faculty  members 
prior  to  and  immediately  following  the  classroom  observations.  The  pre-observation  interview 
was  used  to  establish  important  contextual  features  of  the  observation.  For  example,  we  asked 
where  this  lesson  fit  in  the  overall  structure  of  the  course,  how  much  previous  material  the  class 
had  reviewed  on  this  topic,  and  the  level  of  the  course.  The  post-observation  interview  gath¬ 
ered  data  on  the  faculty  member’s  perception  of  how  well  the  class  went,  the  extent  to  which 
the  faculty  member  covered  the  desired  amount  of  material  at  the  expected  level  of  detail,  the 
strongest  and  weaker  aspects  of  the  class,  and  how  MSP-promoted  practices  were  utilized. 
Each  interview  lasted  approximately  30  minutes. 

Two  researchers  were  on  site  to  conduct  the  observation.  Lessons  were  observed  for  the 
full  class  period.  Within  a  week  of  the  observation,  researchers  completed  an  MSP  Program 
Aims  and  Reflections  Form,  which  was  used  to  describe  the  lesson  along  the  following  pro¬ 
gram  elements:  (1)  lesson  design,  (2)  learning  culture,  and  (3)  lesson  implementation.  These 
notes  were  more  free-form  and  were  used  to  link  what  was  observed  in  the  classroom  to  spe¬ 
cific  MSP  program  aims.  Observers  also  noted  on  this  form  any  unique  or  unexpected  activity 
that  may  have  occurred  in  class.  Field  notes  and  completed  observation  protocols  were  shared 
among  team  members  to  note  any  variation  in  how  the  forms  were  completed  and  to  analyze 
findings. 


MSP  Course  Revision  Questionnaire 

To  better  understand  the  nature  of  the  course  revision  and  its  eventual  impact,  the  MSP  Course 
Revision  Form  was  developed  by  IHE  faculty  in  collaboration  with  the  RAND  research  team 
and  administered  to  the  IHE  faculty  in  the  winter  of  2007-08.  The  form  was  designed  to 
assess  different  aspects  of  the  revised  course,  including  how  the  revisions  changed  the  course, 
the  overall  value  of  the  teacher  fellow  to  the  course  revision,  and  the  anticipated  impact  of  the 
revision  on  student  learning. 
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